Providing binaural sound behind a virtual image being displayed with a wearable electronic device (WED)

ABSTRACT

A wearable electronic device (WED) worn on a head of a user displays a virtual image that has sound. One or more processors process the sound for the virtual image into binaural sound for the user. The binaural sound has a sound localization point (SLP) with a coordinate location that occurs behind the virtual image while the image is located a near-field distance from the head of the user.

BACKGROUND

Three-dimensional (3D) sound localization offers people a wealth of newtechnological avenues to not merely communicate with each other but alsoto communicate more efficiently with electronic devices, softwareprograms, and processes.

As this technology develops, challenges will arise with regard to howsound localization integrates into the modern era. Example embodimentsoffer solutions to some of these challenges and assist in providingtechnological advancements in methods and apparatus using 3D soundlocalization.

SUMMARY

Methods and apparatus improve a user experience during situations inwhich a listener localizes electronically-generated binaural sounds. Thesound is convolved or processed to a location that is behind or near asource of the sound so that the listener perceives the location of thesound as originating from the source of the sound.

Other example embodiments are discussed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a method that improves a user experience while listening tobinaural sound that localizes to a sound source in accordance with anexample embodiment.

FIG. 2 is a method that improves a user experience while listening tobinaural sound that localizes to a sound source in accordance with anexample embodiment.

FIG. 3 is a method that improves a user experience while listening tobinaural sound that localizes to a sound source in accordance with anexample embodiment.

FIG. 4 is a method that detects an action of a portable electronicdevice and changes sound in response to detection of the action inaccordance with an example embodiment.

FIGS. 5A-5H show a listener interacting with a handheld portableelectronic device (HPED) to change sound between being provided inbinaural sound and being provided in mono or stereo sound in accordancewith an example embodiment.

FIGS. 6A and 6B show an electronic or computer system in which alistener hears voices from three different users whose voices localizeexternally to the listener as binaural sound to three different imageson a display of an electronic device in accordance with an exampleembodiment.

FIG. 7 shows an electronic or computer system in which two listeners areengaged in a telephone call in accordance with an example embodiment.

FIGS. 8A and 8B show an electronic or computer system in which alistener sits at a table and engages in a telephone call or other typeof communication with two users via a portable electronic device inaccordance with an example embodiment.

FIG. 9 is a method that selects HRTFs based on a distance of a listenerfrom a sound source in accordance with an example embodiment.

FIG. 10 is a method that convolves, during a telephone call, a voice ofa sound source to a location behind the sound source in accordance withan example embodiment.

FIG. 11 is a computer system or electronic system in accordance with anexample embodiment.

FIG. 12 is a computer system or electronic system in accordance with anexample embodiment.

DETAILED DESCRIPTION

Example embodiments include methods and apparatus that improve a userexperience during a telephone call or other form of electroniccommunication.

One problem with electronically generated binaural sound orthree-dimensional (3D) sound rendering is that listeners have adifficult time determining a location where the sound originates whenthis location is close to the listener. For example, when the locationof the origination of binaural sound is convolved to less than one meterfrom the person (considered to be “near-field”) it may be more difficultfor the person to determine a location or direction of a binaural sound.Whereas, a listener may have more success in external localization whenthe binaural sound is processed to originate from about one meter ormore from the person (considered to be “far-field”).

Head related transfer functions (HRTFs) describe how a sound wavechanges as it interacts with the torso, head, and pinnae of thelistener. HRTFs do not vary substantially in the far-field range. Bycontrast, HRTFs can vary significantly in the near-field range. Suchvariations can be significant depending on the frequency of the sound(e.g., sound below 500 Hz) as frequency at near-field distance has alarge impact on interaural level difference (ILD). Further, modelingnear-field virtual auditory space with HRTFs can be quite complex.

These issues become problematic when an electronic device attempts toconvolve sound so the binaural sound originates to the listener in thenear-field range. The listener may perceive the sound as originatingfrom the wrong location or be unable to localize an origin of the sound.Additionally, convolving sound with near-field HRTFs can becomputationally complex and processor-intensive.

Another problem with electronically generated binaural sound occurs whenthe perceived location of where the sound originates does not match alocation of an image of the sound. For example, a computer or electronicsystem convolves or processes a stream of sound or sound (e.g., a voiceor sound from a game character or object or other virtual sound source)from a video call or telephone software application, computer game, or asocial VR (virtual reality) or AR (augmented reality) softwareapplication. The listener then hears the stream of sound as if the soundis originating from a certain point in space where an image is locatedin the physical or virtual environment around the listener. Theeffectiveness of the user experience is lost on the listener if thelistener is not able to localize or hear the convolved or processedsound at the location of the image. This situation can occur, forexample, when the image is located at a near-field distance from thelistener or the image is occluded or not visible to the listener (e.g.,occluded by a physical or virtual object). At the source of the soundcan be something visible (such as a real or virtual object that thelistener sees) or at the sound source there may be no visual cue orimage (such as a real or virtual object that the listener does not see).As such, instances arise when the location where the listener perceivesan origin of sound does not match an image at the source of the sound ormatch the intended location for the sound. Example embodiments solvethese problems and others in order to improve the user experience of thelistener communicating, navigating, or operating with binaural sound inphysical or virtual space.

Consider an example in which a listener wears a head mounted display orother wearable electronic device and sees a visible source of the soundthat is an image of a caller during a telephone call. The voice of thecaller is not heard to originate from the image shown with the display,but instead originates from a location away from the image. Thissituation would not provide the listener with a credible user experiencesince the voice of the caller is not originating at the image of thecaller.

Binaural sound or 3D audio becomes more realistic when a perceivedlocation of where the sound originates matches or aligns with a visiblesource of the sound, such as an image or object. The realism and userexperience significantly decrease when the location of the originationof the sound does not align with a visible source of the sound.

The problem of aligning a binaural sound with a visible source of thesound becomes exacerbated when the source of the sound is within anear-field distance of the listener. In this situation, even when thesound is convolved with near-field HRTFs in order for the sound tooriginate from the near-field visible source of the sound, the convolvedsound may fail to accurately localize to a listener due to the lowerreliability of externalization from near-field HRTFs relative tofar-field HRTFs.

These problems can also occur when the source of the sound is notvisible but the listener knows a location of the source of sound. Forexample, a listener sees and talks with another person and thenmomentarily looks away in another direction.

During the moment of looking away, the listener does not see the otherperson, but knows a precise location or area where he or she is located.The listener expects a voice from the other person to originate from thelocation.

One way to convolve binaural sound is to convolve the sound to alocation where the sound source is visually perceived (e.g., convolvethe sound to the coordinate location of the image, virtual sound source(VSS), avatar, or display). In some instances, it may be difficult,undesirable, or impossible to convolve the sound to the visuallyperceived location of the sound source. For example, the electronicdevice convolving the sound is not able to select HRTFs with coordinatesthat match coordinates of the sound source or a location of the soundsource as presented to the listener. Such HRTFs may not exist. Further,selecting or deriving such HRTFs may be process-intensive or haveanother disadvantage. So where should sound be convolved to localize inthese instances? What HRTFs should be selected, if any?

Example embodiments solve these problems and improve the user experienceof a listener externally localizing binaural sound.

Electronic devices with example embodiments displaying the sound sourceor providing sound may not include speakers to generate sound or mayinclude speakers but the electronic device does not generate sound withthe speakers.

Instead, a listener may receive the sound from a handheld or portableelectronic device, such as headphones, earphones, earbuds, a wearableelectronic device, an electronic device that provides sound from boneconduction, or another type of electronic device that provides binauralsound to a listener. Further, the electronic devices may includewireless and wired electronic devices, such as wireless earphones,wireless headphones, electronic glasses with earphones or earbuds, ahead mounted display (HMD) or optical head mounted display (OHMD) withearphones or earbuds, a smartphone with wireless earphones or earbuds,etc.

Electronic devices of example embodiments are not required to emit soundto a listener in a conventional manner (e.g., with speakers at theelectronic device that transmit sound through the air to the listener).For example, an electronic device is a smartphone, computer, television,display, or other electronic device with speakers muted, decoupled, orpowered off. Nevertheless, it may be useful or convenient for a listenerto localize sound at the electronic device. For instance, binaural soundlocalizes to an image displayed on or through the display of theelectronic device when such an electronic device is not providing soundto the listener in the conventional manner.

Although sound may not emanate from speakers of an electronic device ina conventional way, example embodiments process the sound to appear tothe listener to originate or emanate from a sound source or particularlocation (e.g., at the electronic device, behind the electronic device,in front of the electronic device, to a side of the electronic device,or to another location). The binaural sound causes the listener toperceive the sound as originating from or localizing at the sound sourceeven though the sound source emits no sound through the air. Instead,the listener hears the convolved binaural sound through earphones,earbuds, headphones, wearable electronic device, or another apparatus.

One or more processors (such as a digital signal processor or DSP)processes or convolves the sound to a location that is not coincidentwith or not at the location of the sound source. For example, a DSPprocesses the sound to localize to a coordinate location that does notmatch a coordinate location of the sound source. Nevertheless, thelistener hears the sound as originating from or believes that the soundoriginates from the sound source. Sound originates to the listener at asound localization point (SLP) that is coincident with or at the soundsource. The SLP experienced by the listener is influenced but notdictated by the HRTF pair convolving the sound. Likewise a visual imagethat a listener associates with an auditory event also influences theSLP. For example, a sound of a dog barking is convolved to a location infront of a listener at a distance of three meters, and the sound isstored in a file. The sound file is played to a blindfolded listenerwho, externalizing the binaural sound, approximates that a dog is threemeters away. The distance of the SLP of the barking sound is threemeters. Consider a listener without a blindfold, also hearing the soundfile of the barking dog convolved to the distance of three meters, andin addition, seeing an image of a barking dog in front of him or her twometers away. In this case the listener, seeing the image of the dogassociated with the auditory event of the sound of the dog, localizesthe barking sound to the associated sound source (at two meters).Although a single sound file is played, the SLP for the seeing listeneris two meters from him or her, whereas the SLP for the blindfoldedlistener is three meters from him or her. Thus a sound convolved with acertain pair of HRTFs is externalized at different SLPs depending on thecircumstances, situation, and/or listener.

Example embodiments include methods and apparatus that help to solvethese problems and other problems with binaural sound. Exampleembodiments improve realism and the user experience of a listenerlistening to binaural sound when the sound source or source of the soundis close to the listener, such as being in a near-field range from thelistener. Further, example embodiments are applicable to both visibleand non-visible sound sources.

One or more of several different techniques of executing hardware and/orsoftware convolve the sound so the listener believes that the soundoriginates from the sound source when in fact the sound is convolved toa different location. These techniques include placing or positioningthe origination point of the sound on a line-of-sight with the soundsource, placing the origination of the sound on an imaginary line fromthe head of the listener (e.g., a point between the ears of thelistener) to the sound source, placing the origination of the soundbehind the sound source, simulating the direction of the sound as thoughcoming from the direction of the sound source, adjusting a volume of thesound to be commensurate with the distance to the sound source, andexecuting other example embodiments discussed herein.

FIG. 1 is a method that improves a user experience while listening tobinaural sound that localizes to a sound source.

Block 100 states display, with an electronic device, a sound source to aperson.

In an example embodiment, the sound source is visible to the person,such as being in a field-of-view or line-of-sight of the person. Forinstance, the listener sees an image on, through, or with a display ofan electronic device. Alternatively, the sound source is not visible tothe person, such as not being in a field-of-view or line-of-sight of theperson. For instance, an image or a virtual object is shown on anelectronic display as behind the listener, in another room, or presentlynot visible. As another example, the electronic display is behind thelistener, in another room, or presently not visible.

By way of example, the electronic device is a smartphone, a tabletcomputer, a notebook or laptop computer, a desktop computer, atelevision, a display, an electronic watch, a screen, a portableelectronic device (PED), a handheld portable electronic device (HPED), anon-portable computer or non-portable electronic device. For instance,the electronic device is a smartphone or tablet computer held in a handof the person, a laptop computer or tablet computer or desktop computerresting on a surface in front of the person, an electronic watch with adisplay on a wrist of the person, a portable electronic device on a lapof the person while the person sits in a chair or passenger seat, atelevision in a home, store, or other public place, a home appliance, adisplay mounted on or affixed to an object (such as a wall ordashboard), or an advertising display (such as an edge display orcounter display or other electronic device that provides advertisementsto people).

For example, the sound source is an image, such as a video, a picture,an AR image (e.g., an image of a virtual sound source), a VR image or acombination of one or more of these images. For instance, the soundsource is a real-time video of a person captured with a camera anddisplayed to the person, an AR image displayed with a wearableelectronic device (e.g., electronic glasses or a smartphone mounted to awearable head mount), a VR image displayed with a head mounted display(HMD), a two-dimensional (2D) image on a flat display (e.g., a TV or anadvertisement display), a 3D image, a stationary image, a moving image,a recorded image, a real-time image, a hologram, or another image thatis generated, reproduced, or displayed with an electronic device.

Block 110 states improve the user experience of the person by processingsound of the sound source with sound localization information (SLI) thatpositions the sound to a location that is located behind a location ofthe sound source so the person perceives an origin of the sound asoriginating from the location at the sound source.

The position to where the sound is convolved is not the location of thesound source. Instead, the location where the sound is convolved isbehind the sound source from the point-of-view of the listener.

As noted, in some instances, it may be difficult, undesirable, orimpossible to convolve sound to the location of the sound source. Inthese instances, an example embodiment selects HRTF pairs havingcoordinates of a position behind the sound source.

By way of example, the sound source is a display of a handheld portableelectronic device (HPED) that the listener holds (e.g., a smartphone ortablet computer). The HPED displays an image of a talking person whilethe display is one to two feet in front of a face of the listener. TheHPED selects a HRTF pair that has coordinates in a far-field range about1.0 meter from the face of the listener. A processor in the HPEDprocesses a voice of the talking person so the voice localizes to thecoordinate location of the HRTF pair. The coordinate location is infront of the face of the listener, directly behind the image of thetalking person on the display of the HPED, and located about 1.0 meteraway from the face of the talking person. The listener hears or believesthat the voice of the talking person originates from the image of thetalking person on the display that is one to two feet away. Thissituation occurs even though the display is closer to the face of thelistener than the selected coordinate location of the HRTFs beingexecuted to convolve the voice of the talking person.

The listener thus perceives that the sound originates from the imageeven though the sound is convolved to originate from a position that isbehind the image. This situation occurs because listeners are accustomedto hearing sound originate from a physical auditory event, such as at aphysical object. Listeners consciously or unconsciously try to visuallylocate an object or event that corresponds to the sound and to associateor to place the origination of the sound at the object. Thus, eventhough the sound does not actually originate from the image, the brainignores or reconciles a difference of position, resulting in the originof the sound being localized at the image. When coordinates of thelocalization point occur beyond and behind the image, the soundlocalizes to the listener at the image. This situation also occursbecause listeners have a difficult time in accurately judging exact orprecise distances to an origin of sound, especially when the sound isnot a voice and/or when the source of the sound is not visible to thelistener. One or more example embodiments take advantage of theseobservations and trick the listener into believing that the soundoriginates from the sound source when in fact the sound is convolved tooriginate from a location that is behind the sound source.

Consider an example in which a processor processes sound with HRTFshaving coordinates of a far-field location even though the location ofthe source of the sound (e.g., an image being displayed) is a near-fielddistance from the listener. In this example, the sound originates frombehind the source of the sound. During this time, however, the listenerperceives the sound as originating from the location at the source ofthe sound and not behind the source of the sound.

Consider an example embodiment that improves a user experience when anelectronic device, an image, or an object is a source of sound that islocated within a near-field distance from the person (e.g., less thanone meter away from a face of the person). A processor processes orconvolves the sound with far-field HRTFs so the sound originates to alistener beyond yet behind the electronic device, the image, or theobject. Nevertheless, the listener believes that the sound originatesfrom the location of the electronic device, the image, or the object.

In an example embodiment, a method improves a user experience during avideo call or a telephone call between a person and another person whenthe person uses a HPED that is located within a near-field distance fromthe person. For instance, the person holds the HPED in front of his orher face to see a real-time video or image of the other person. Forexample, a camera captures video of the other person, and the personsees the video or an AR or VR image based on the video. One or moreprocessors convolve a voice of the other person with far-field HRTFs toa coordinate location in empty space behind the HPED located at thenear-field distance. The person would otherwise localize the voice ofthe other person at the coordinate location of the HRTFs in empty space.However, the person perceives the voice of the other person asoriginating from the HPED when the HPED is located between the face ofthe person and the location where the voice of the other person isconvolved to the person. Thus, the person hears the sound as originatingfrom the HPED at a near-field distance even though the coordinatelocation of the HRTFs is located behind the HPED at a far-fielddistance.

Consider an example embodiment in which the sound source is at an HPEDor smartwatch and not an image of a person; and the sound at the soundsource is a chime or alert sound. For instance, a listener wearsheadphones that are coupled to multiple electronic devices that arewithin near-field and/or far-field distances. When the listener hears analert sound, music, or announcement issued by one of the electronicdevices, the example embodiment convolves the sound to a far-field pointon the line extending from his or her head toward the electronic devicethat issued the sound. Processing the sound in this manner allows thelistener to monitor which one of the electronic devices issues thesound.

Consider an example embodiment that convolves or processes sound so thecoordinates to where the sound is located occur behind an image that isthe source of sound to the listener. For example, a head of a listeneris located at an origin with spherical coordinates of (0, 0, 0), and anelectronic device with a display is located at (0.5 m, 20°, 20°) withrespect to a forward-looking direction of the listener. In order todeliver binaural sound that will be perceived as originating from thedisplay of the electronic device, sound could be convolved with a pairof HRTFs corresponding to the location of the electronic device (0.5 m,20°, 20°). The location of the electronic device, however, is anear-field location with respect to the location of the listener. HRTFswith the near-field location may not be known, may be tooprocess-intensive to convolve with a processor, or may fail to generatereliable audial cues to result in a sound localization point (SLP) forthe listener. Instead of processing the sound with the near-field HRTFs,the electronic device selects HRTFs having coordinates that are locatedat a far-field distance with respect to the listener. For example, theelectronic device selects different HRTFs with spherical coordinates of(1.2 m, 20°, 20°). The coordinate location of the different HRTFs occurdirectly behind a center of the electronic device. Specifically, adistance from the listener to the location to which the different HRTFscorrespond is 1.2 m, whereas a distance from the listener to thelocation of the electronic device is 0.5 m. Further, the location of thecoordinates of the different HRTF pair and the location of thecoordinates of the electronic device occur on a straight line to acenter of the head of the listener (e.g., inside the head of thelistener and between his or her ears). Processing the sound with thedifferent HRTFs at the far-field location (as opposed to the near-fieldlocation) is less process-intensive and improves the chances of thelistener localizing the sound at the sound source. Further, since thelocation of the coordinate location of the different HRTFs occurdirectly behind the coordinate location of the electronic device, thelistener perceives the sound as originating from the electronic deviceand not from behind the electronic device.

An example embodiment convolves sound with HRTFs having coordinates thatare behind the source of the sound. A distance from the listener to thelocation of the coordinates of the HRTFs is greater than a distance fromthe listener to the source of the sound. For example, a processorconvolves or processes the sound to originate at a coordinate locationthat occurs behind a coordinate location of the visible source of thesound. In some instances, the visible source of the sound obstructs aview of the location to where the sound is convolved. For instance, ifan image appears on a display and the sound originates from a locationbehind the display, the listener would not be able to see the locationto where the sound is convolved since the display blocks the listenerfrom seeing the location.

Consider an example in which the source of the sound is an electronicdevice with a display, and the listener sees an image as the visiblesource of the sound on or through the display. For example, the listenersees a real-time video, a picture, an augmented reality (AR) image, avirtual reality (VR) image, or another object or image that the listenerattributes to a sound source. The electronic device, however, is closerthan one meter to a face of the listener and, as such, is within anear-field distance from the listener. Instead of convolving the soundwith near-field HRTFs, the electronic device convolves the sound withfar-field HRTFs. The electronic device selects coordinates of thesefar-field HRTFs so the location of where the listener hears the soundcoming from is behind the electronic device. The location is on aline-of-sight or an imaginary line that extends from the head of thelistener to a location where the image is shown on or through thedisplay. The listener observes that the binaural sound originates fromthe image even though the sound is convolved to a location that isbehind the image.

FIG. 2 is a method that improves a user experience while listening tobinaural sound that localizes to the sound source.

Block 200 states display, with an electronic device, a sound source to aperson.

The electronic device provides or presents the sound source to theperson, such as displaying an image on, with, or through a display ofthe electronic device (see block 100 for further examples).

Block 210 states improve the user experience of the person by processingsound of the sound source with sound localization information (SLI) thatpositions the sound to a location that is on a line-of-sight from theperson to the sound source so the person perceives an origin of thesound as originating from the location of the sound source.

The location where the sound is convolved or positioned is not thelocation of the sound source. Instead, the location where the sound isconvolved is before the sound source or after the sound source and onthe line-of-sight from the listener to the sound source.

As noted, in some instances, it may be difficult, undesirable, orimpossible to convolve sound to the location of the sound source. Inthese instances, an example embodiment selects HRTFs with coordinatesthat exist on an imaginary line that extends from a head of the listenerthrough the sound source. For instance, the imaginary line exists as aline-of-sight when a head of the listener is facing the sound source andthe eyes of the listener are looking toward the sound source. As anotherexample, the line extends from a location inside the head of thelistener (e.g., between the eyes of the listener) to an image or objectrepresenting the sound source. In this instance, the listener is notrequired to be looking at the sound source, and hence calculations basedon a direction where the eyes of the listener are looking are notrequired. Consider an example in which the listener knows the locationor direction of the sound source without actively seeing the soundsource (e.g., the listener has already seen the sound source in anearlier field-of-view, the listener anticipates the location of thesound source, the listener has not and/or does not see the sound sourcebut was or is informed of the location of the sound source in anotherway). The sound is convolved upon an imaginary line in the directioncoinciding with the sound source, and hearing the sound positioned onthe line produces the perception of the sound originating at the knownposition of the sound source that is also on the line.

When sound is convolved to originate on or near the line, then thelistener is more likely to understand or to comprehend that the sound isattributed to or originates from the position of the sound source. Thissituation occurs when the sound is convolved to localize on or near theline in front of the sound source or behind the sound source. Thus, twoseparate optional areas exist that are not located at the location ofthe sound source and for which HRTF pairs of associated coordinates maybe available to execute convolution of the sound. One option is toselect HRTFs with coordinates on the line in front of or before thesound source. In this option, the location of the coordinates of theHRTFs is between the face or head of the listener and the sound source.Another option is to select HRTFs with coordinates on the line behind orafter the sound source. In this option, the sound source is between theface or head of the listener and the location of the coordinates of theHRTFs.

A processor processes or convolves the sound to originate at a locationthat is on an imaginary line-of-sight that extends from the listener tothe source of the sound. Consider an example in which the source ofsound is an image displayed on a display of an electronic device (e.g.,a display of a smartphone) or displayed as a VR or AR image with awearable electronic device. A coordinate location of where the sound isconvolved on the line-of-sight is located before the image, at theimage, and/or behind the image. As noted, selecting the coordinatelocation at the image may not be possible, efficient, or desirable. Asan alternative, example embodiments select coordinate location(s) beforeand/or after the image and on the line-of-sight. The sound of the soundsource may be convolved to multiple locations along the linesimultaneously, emphasizing or highlighting the sound. Highlighting thesound improves the experience of the listener in establishing thedirection of the localization and/or identifying the sound source towhich the sound is attributed.

An example embodiment convolves the sound to varied locations on theline during the playing of the sound. During the playing of the sound,the example embodiment monitors the response of the listener to thelocalization in order to establish a best or better location thatimproves the accuracy of the localization and experience of thelistener. For example, for any particular sound that is convolved tolocalize at (r, θ, φ) on the line extending from the head of thelistener, there is an optimal distance R that improves the experience ofbinaural sound for each listener and each sound source. For example, forsome sounds, sound sources or images, listeners, and directions (θ, φ),the optimal value of R may be larger (e.g., improving the positionalrealism of the experience of the listener in establishing the directionof the localization for sound sources that require accuracy of position)or may be smaller (e.g., increasing the intelligibility or volume of thesound for sound that carries more data such as speech). An exampleembodiment varies the distance r and gathers active or passive feedbackfrom the listener in order to further adjust r to find the optimal R foreach sound source. Finding an optimal R for each source of soundimproves the effectiveness of binaural sound in delivering data and/orrealism to the listener.

The sound can also be convolved to originate from locations adjacent toor near the line-of-sight. With some sounds, a listener is not able toaudibly distinguish between a sound originating exactly from the visiblesource of the sound (e.g., exactly along the line-of-sight to the image)and a sound originating from a location adjacent to the image oradjacent to the line-of-sight. Sound convolved to a coordinate locationsufficiently close to a visible source of the sound can cause thelistener to localize the sound at the sound source. Success in alteringa localization of a listener depends on several factors, such asdistance between the listener and the source of the sound, a loudness ofthe sound, a type of sound (e.g., whether the sound is familiar orunfamiliar, voice or non-voice), a size of the source of the sound,whether other noise or other sounds are present, and other factors. Forexample, sound is convolved to originate on the line-of-sight or withina predetermined distance from the line-of-sight.

Consider an example in which a display in an apartment store shows videoadvertisements. The display has no speakers. Instead, a soundlocalization system (SLS) wirelessly transmits binaural sound tolisteners through electronic earbuds, earphones, or headphones that thelisteners wear. As listeners walk near the display, they hear binauralsound originate from the display. In some instances as the listenerswalk by, the SLS does not have exact coordinates to convolve sound tooriginate from a middle or center of the display (for example, the SLSdoes not have HRTF coordinates corresponding to a center position of thedisplay for a known head orientation of a listener, or an accurate headorientation is unknown). In this situation, the SLS localizes the soundso it appears to originate next to the display, such as on an edge ofthe display or a few inches from the display. When the listener sees thedisplay, the listener localizes the sound as originating from thelocation of the display since the audial cues of the binaural soundsuggest to the listener a location next to or close to the display.

A volume of the sound can also be adjusted to assist a listener inlocalizing binaural sound to the sound source when the sound isconvolved to a location other than the sound source. For example, anexample embodiment convolves sound to have an adjusted volume level thatcorresponds to the distance from the listener to the visible source ofthe sound even when the distance to where the sound is convolved isdifferent (such as a distance reaching beyond the visible source of thesound). Adjusting the volume level assists the listener in localizingthe sound at the visible source of the sound and not at a differentdistance.

Consider an example in which a listener telephones a friend on asmartphone that displays a real-time video of the friend during thetelephone call. The listener holds the smartphone about one foot awayfrom his face while he talks to his friend and watches the video of hisfriend. An image of the friend on the display of the smartphone is largeand is proportioned to represent a size of the friend as if the friendwere standing one foot from the face of the listener. Even though thesmartphone is within a near-field distance from the face of thelistener, the smartphone executes HRTFs with a far-field location andconvolves the voice of the friend to a location that is four feet awayfrom the face of the listener. The location is behind the smartphone andalong a line-of-sight from the listener to the smartphone. Thesmartphone further adjusts a volume of the voice of the friend tocoincide with a distance of one foot in front of the face of thelistener, not four feet from the face of the listener. The volume thuscorresponds to a distance from the listener to the image of the friendand not a distance from the listener to the coordinates to where thesmartphone convolves the sound.

FIG. 3 is a method that improves a user experience while listening tobinaural sound that localizes to a sound source.

Block 300 states display, with an electronic device, two or more soundsources to a person.

The electronic device provides or presents the sound sources to theperson, such as displaying images on, with, or through a display of theelectronic device (see block 100 for further examples).

In an example embodiment, a display of the electronic device is divided,partitioned, or split into two or more portions or windows. Each portionincludes an image of one of the sound sources.

Block 310 states improve the user experience of the person by processingsound of one or more of the sound sources with sound localizationinformation (SLI) that positions the sound to a location that is behindthe one or more sound sources and at an angle with respect to aline-of-sight from the person to the one or more sound sources so theperson perceives an origin of the sound as originating from the locationof the one or more sound sources.

Binaural sound includes audial cues to cause a listener to localize thesound to a location that is offset from the line-of-sight of thelistener to the sound source. For example in spherical coordinates, theoffset has an azimuth angle (θ) and/or an elevation angle (φ).

Consider an example in which a listener talks with an intelligent useragent (IUA), intelligent personal assistant (IPA), or other computerprogram on an electronic device with a display. The displaysimultaneously shows an image representing the IPA or data provided bythe IPA and an image of the listener. The listener faces the displaywith a forward-looking direction between the two images. For instance inspherical coordinates, the listener is located at (0, 0, 0), and acenter of the display is located at (0.3 m, 0, 0). An image of the IPAis on a right side of the listener at +θ₁, and an image of the listeneris on a left side of the listener at −θ₂ while the listener looksstraight ahead with the forward-looking direction. An example embodimentpositions or convolves a voice of the IPA to a position behind theelectronic device at an angle with a positive azimuth coordinate. Forinstance, if the forward-looking direction has an azimuth coordinate ofθ=0°, then the voice of the IPA is convolved with coordinates within arange of 0°<θ<30° (such as θ=5°, 10°, 15°, 20°, 25°, or) 30°. Further,the voice of the IPA is convolved to coordinates that occur behind thedisplay with a distance (r)>0.3 m. For instance, the voice of the IPA isconvolved with HRTF pairs corresponding to spherical coordinates locatedin the following range: r>1.0 m, 0°<θ<30°, and φ=0°.

One problem in the technological field of binaural sound is that alistener may need to quickly change between listening to sound asbinaural sound and listening to the sound as mono sound or stereo sound.

One or more example embodiments solve this problem and provide a quickand convenient way for a listener to change between listening to soundas binaural sound and listening to the sound as mono sound or stereosound so as to discontinue externalization of the sound.

FIG. 4 is a method that detects an action of a portable electronicdevice and changes sound in response to detection of the action.

Block 400 states detect an action of a portable electronic device.

The actions include, but are not limited to, one or more of rotating theportable electronic device, moving of the portable electronic device(e.g., shaking the portable electronic device or moving it through theair in a predetermined way), gripping or holding the portable electronicdevice (e.g., grabbing the portable electronic device with a hand),activity of a person (e.g., sensing when the person is walking orrunning or sitting), releasing the portable electronic device (e.g.,releasing the portable electronic device from a hand), covering a sensorof the portable electronic device (e.g., covering or darkening a lens ofa camera), detecting a face of a person (e.g., detecting with facialrecognition software the presence or proximity of a person), detectingabsences of a face of a person (e.g., detecting with facial recognitionsoftware an absence of a face of a person), detecting or sensing light,detecting or sensing darkness, detecting or sensing presence of a person(e.g., with a sensor), detecting or sensing an identity or biometric ofa person (e.g., detecting a fingerprint or thumbprint of the person,identifying a person by iris image, retina scan, ear form, or anotherbiometric), detecting a change in an electrical power source of a PED(e.g., changing between battery-supplied power and another source ofpower), detecting a change in audio output configuring of a PED (e.g.,changing from between sound being output from speaker and sound beingoutput from headphones, detecting the an event of headphones beingplugged-in or unplugged, coupled to the PED or decoupled, powered on oroff), or another action discussed herein.

By way of example, one or more sensors in the portable electronic devicedetects when the action occurs. For instance, these sensors include, butare not limited to, a camera, a gyroscope, an accelerometer, amagnetometer, a compass, an optical or capacitive scanner, a display, aproximity sensor, a light sensor, a pedometer, a fingerprint sensor, oranother sensor.

Block 410 states change, in response to detecting the action, soundbeing provided to a listener (1) from being binaural sound to being monosound or stereo sound or (2) from being mono sound or stereo sound tobeing binaural sound.

The portable electronic device changes or switches sound being providedto the listener in response to detecting the action. Consider an examplein which a three-axes accelerometer and/or gyroscope in the portableelectronic device senses rotation of the portable electronic device. Inresponse to detecting the rotation, the portable electronic device takesan action with regard to the sound. These actions include changing orswitching the sound (1) from being binaural sound to being mono sound orstereo sound or (2) from being mono sound or stereo sound to beingbinaural sound. Other actions include, but are not limited to, mutingthe sound, lowering the volume, raising the volume, stopping the sound,ending or terminating a telephone call, placing a telephone call onhold, joining another call, joining another virtual auditory space,muting or pausing or changing a particular sound type or input source,or performing another action.

In an example embodiment, changing from binaural sound to mono sound orstereo sound or changing from mono sound or stereo sound to binauralsound occurs when the portable electronic device detects one or more ofa predetermined amount or degree of rotation, a predetermined speed ofrotation, and/or rotation in a particular direction or about aparticular axis.

Consider an example in which the portable electronic device includes aninertial motion unit (IMU) with an accelerometer, magnetometer, andgyroscope. The IMU senses or detects an amount and/or speed of rotationof the portable electronic device. When the amount of rotation and/orspeed of rotation reaches a predetermined threshold, the portableelectronic device executes one or more of the actions (e.g., changes thesound from binaural to mono or from mono to binaural).

For example, a listener holds a smartphone in his or her hand androtates the smartphone by a threshold value of ninety degrees (90°) tochange the sound from binaural sound to mono sound. When the listenerrotates the phone back (e.g., −90°), the sound changes from mono soundback to binaural sound. Rotation of the smartphone by the thresholdvalue while in the hand of the listener thus provides a convenient wayto change between listening to sound in binaural sound and listening tothe sound in stereo or mono sound.

Example embodiments are not limited to the threshold value or amount ofninety degrees (90°) since other threshold values or amounts can bedesignated to trigger a change between binaural sound and mono or stereosound. By way of example, other threshold values include, but are notlimited to, one or more of twenty-five degrees (25°), thirty degrees(30°), thirty-five degrees (35°), forty degrees (40°), forty-fivedegrees (45°), fifty degrees (50°), fifty-five degrees (55°), sixtydegrees (60°), sixty-five degrees (65°), seventy degrees (70°),seventy-five degrees (75°), eighty degrees (80°), eighty-five degrees(85°), ninety degrees (90°), ninety-five degrees (95°), etc.

Consider an example in which a camera in a HPED detects a left and/orright facial profile of a first user during a telephone call or othercommunication with a second user. An example embodiment positions orrepositions where the voice of the second user localizes to the firstuser in response to detecting the right and/or left facial profile. Forexample, when the HPED detects a right profile of the listener, thedetection triggers a change in the convolution of the voice to alocation with an azimuth coordinate (0) of greater than positive seventydegrees (+70°) and less than positive one hundred degrees (+100°), suchthat +70°≤θ≤+100° with respect to a line-of-sight of the first user.

Consider an example in which the action of changing sound betweenbinaural sound and mono or stereo sound occurs when the electronicdevice detects a change in orientation of the electronic device withrespect to a face and/or head of the user. When the change reaches orexceeds a threshold value, then execute the change. The change inorientation can occur in one of three ways. First, the orientation ofthe head and/or body of the user changes with respect to the electronicdevice while the electronic device does not move. For instance, a userrotates his head or moves with respect to a stationary camera, facingsensor (e.g., front-facing sensor, rear-facing sensor), or other sensor.Second, the orientation of the electronic device changes with respect tothe user while the user does not move. For instance, a user holds asmartphone in his or her hand while the camera captures an image of theface and executes facial recognition and distance determination. Theuser rotates the smartphone so the camera no longer captures the face ofthe user. Third, both the user and the electronic device move to changethe orientation of the user with respect to the electronic device. Forinstance, the user holds the smartphone in his hand and simultaneouslyrotates his head and the hand holding the smartphone.

Consider an example in which the action of changing sound from binauralto mono/stereo or from mono/stereo to binaural is triggered when a PEDdetects rotation of a particular rotational or angular speed of the PED.For example, a listener holds a smartphone in his or her hand androtates the smartphone slowly and no change is triggered. When thelistener rotates the smartphone quickly, a change is triggered. Thechange is triggered when the rotation occurs within a predetermined timeor crosses another threshold, such as a predetermined number ofrevolutions per second or radians per second. By way of example, achange in the sound is triggered when a quarter rotation or ninetydegrees of rotation occurs within a range of 0.1 seconds-0.5 seconds.

Consider an example in which a listener holds a smartphone in front ofhis face while talking to a friend. The listener sees an image of hisfriend, and a camera in the smartphone captures an image of the listenerthat transmits to the friend during the telephone call. A processor (inthe smartphone or in a cloud server) processes the voice of the friendin order to be localized as binaural sound at the location of thesmartphone in the hand of the listener. When the listener rotates thesmartphone ninety degrees downward so the display faces the ground, thevoice of the friend changes from being provided to the listener inbinaural sound to being provided to the listener in mono sound. When thelistener rotates the near edge of the smartphone ninety degrees upwardlyso the display again faces the listener, the voice of the friend changesfrom being provided to the listener in mono sound to being provided tothe listener in binaural sound.

Consider an example in which a listener holds a smartphone in front ofhis face while talking to a friend. The listener hears the voice of thefriend as binaural sound that localizes to an image on the display ofthe smartphone. To switch the voice from being provided in binauralsound that externally localizes to the smartphone to being provided inmono or stereo sound that localizes inside a head of the listener, thelistener performs one of the actions discussed herein. As one example,the smartphone changes the sound when a camera in the smartphone ceasesor fails to detect a face of the listener. For instance, providebinaural sound only while the camera detects the face of the listener.Change to mono sound or stereo sound when the camera no longer detectsthe face of the listener. As another example, the smartphone changes thesound when the camera or another sensor detects darkness. For instance,provide binaural sound if the camera or sensor detects a certain levelof light. Change to mono sound or stereo sound when the camera or sensordetects an obstruction or a certain drop in the level of light (e.g.,the listener puts his or her hand over the sensor or camera; thelistener places the smartphone on the table, covering the display orcamera; the listener obstructs the display or sensor by placing thesmartphone against his or her body, etc.).

Consider an example in which an electronic device captures an image of aperson with a camera. The electronic device executes a softwareapplication to detect a face and perform facial recognition. During theperiod of time that the camera detects the face of the person, theelectronic device provides sounds to the person as binaural sound. Whenthe camera no longer detects the face of the person, then the electronicdevice performs an action (such as changing the sound from binauralsound to mono sound or stereo sound, terminating external localizationof the sound by the person).

Facial detection and/or facial recognition enables a person to changequickly back and forth between external localization of binaural soundand internal localization of mono or stereo sound. For example, during atelephone call or while playing a software game, a listener holds a HPED(e.g., holds a smartphone in his or her hand) or wears a WED (e.g.,wears an electronic watch). A camera in the HPED or WED monitors theface of the listener. When the listener desires to change to mono soundor stereo sound, the listener rotates the HPED or WED so the cameraceases to capture or detect the face of the listener. When the listenerdesires to change back to externalizing the sound, the listener rotatesthe HPED or WED so that the camera captures or detects the face of thelistener.

Instead of or in addition to rotating the HPED or WED, an exampleembodiment allows the listener to change the sound between binaural andmono and/or stereo by moving his or her head. A camera in the HPED orWED monitors the face of the listener. The HPED or WED changes the soundwhen the HPED or WED detects a change in head orientation or a change ingaze of the listener (e.g., the listener looks away from the camera ordisplay or rotates his or her head by a predetermined amount). Forinstance, the electronic device changes from binaural sound to mono orstereo sound when one of the following occurs and/or changes from monoor stereo sound to binaural sound when one of the following occurs: thelistener stops gazing or looking at the display, the listener turns hisor her head away from the display, the camera no longer detects the faceof the listener, the camera detects a “full face” view, a side profile,a “three-quarter” view, or another view between a “full face” view and aprofile of a face of the listener, the camera detects a top of the headof the listener, and the camera detects that the eyes of the listenerare closed for a predetermined amount of time.

Consider an example in which two people (e.g., a first person and asecond person) talk to each other during a telephone call. Their voicesexternally localize to each other as binaural sound to images ondisplays of their respective HPEDs. A camera or facing sensor in theHPED captures and/or detects a face of the person. Convolution isadjusted in response to sensing a change in orientation of a head of aperson, such as a change in a profile angle of a face of a person. Forexample, when the face of the first person is oriented toward thedisplay, then convolve the voice of the second person as binaural soundto the location of or behind the display. When the face of the firstperson moves to a position exposing a right facial profile to a displayof the HPED, the voice of the second person moves. For instance, theHPED moves the voice of the second person from convolving the voice toan azimuth direction (θ) of zero degrees (0°) to convolving the voice toan azimuth direction of positive ninety degrees (+90°) with respect to afacing direction of the head of the first person. When the orientationof the head of the first person relative to a display of the HPEDchanges to a second relative orientation such that a left facial profilefaces a display of the HPED, the voice of the second person moves. Forinstance, the HPED adjusts the convolution of the voice of the secondperson such that the first person localizes the voice at a location withan azimuth coordinate (θ) of negative ninety degrees (−90°) with respectto the facing direction of the head of the first person. As theelectronic device captures and detects a change in the facialorientation, the example embodiment adjusts the convolution of the voiceof the second person so that the first person continues to localize thevoice at the electronic device.

An example embodiment improves the user experience of a listener byenabling the listener to find an electronic device, or identify which ofmultiple electronic devices is the sound source of binaural sound thatthe listener is hearing and localizing. Consider the example above inwhich the displays are turned off during a phone call so that images arenot displayed at the HPEDs. During the phone call, the electronicdevices function as the sound sources rather than the images displayedon or with the electronic devices. The example embodiment determines theorientation of the head of the listener and convolves the sound (here, avoice) to a position on a line extending from the center of the head ofthe person to and through the HPED of the person. When the orientationof the HPED changes relative to the head of the person receiving thesound, the voice is convolved to a position on a new line extending fromthe head of the person to and through the HPED of the person. Consideran example where an electronic device monitors the orientation of a headof a listener, and the measurement of the orientation of the head iscommunicated to or available to other electronic devices that do notmonitor head orientation directly.

Consider an example in which a listener wears wireless earphones whilewalking in a department store. When the listener gazes at a display, acamera detects the face and gaze of the listener. In response todetecting the face, the display plays a content-specific advertisementto the listener. Binaural sound from the advertisement is convolved suchthat the listener will externally localize the sound at the display.When the listener turns his or her head, the camera detects a change inthe orientation of the head of the listener and that the forward lookingdirection of the listener no longer intersects the display and stopsplaying the binaural sound to the listener.

Sound can also change between binaural sound and mono or stereo soundwhen the electronic device detects darkness, detects light, or detectsblockage of a sensor. For example, in response to detecting when a lensof a camera is blocked, cease providing sound to a person in binauralsound and commence providing the sound to the person in mono or stereosound.

Consider an example in which a listener talks to another person during atelephone call while holding a smartphone or other HPED. The listenerhears the voice of the other person as binaural sound that localizes ator on the other side of the HPED. The HPED changes to mono or stereosound upon detecting an action from the user, such as detecting blockageof the camera (e.g., a finger on the lens of the camera), detecting afingerprint or thumbprint, detecting a finger or thumb on the display,detecting a facial gesture or hand gesture, or detecting another action.Detection of one of these actions enables the listener to change quicklybetween binaural sound and mono or stereo sound.

Changing between binaural sound and mono or stereo sound can also occurin response to detecting or sensing a change in distance of a listenerfrom an object, such as the electronic device. For example, a camera orfacing sensor in a HPED tracks an image or orientation of a person orface of a person during a telephone call. The person hears the voice ofthe other person during the telephone call in binaural sound. The HPEDautomatically switches or changes the voice from externally localizingin binaural sound to localizing in one of mono or stereo sound when theperson moves a predetermined distance from the HPED. For instance,change the sound when the person moves more than one meter away from apresent position or from the HPED, more than 1.5 meters away, more than2.0 meters away, etc. A camera or sensor captures an image ororientation of the head of the person, and the distance of the person orhead from the HPED is determined based on a size of the face and/or bodyimage captured or sensed with the camera or sensor.

Consider an example embodiment that monitors states of a PED providingsound to a listener and changes the sound in response to change in astate. For example, a listener without headphones hears sound from asingle speaker included in the body of a PED. The listener couples orplugs-in headphones to the PED for private listening. Alternatively, thelistener commences to listen with wireless headphones that communicatewith the PED. The example embodiment detects the change in the audiooutput device from the speaker to the headphones. The detection of thechange of output device triggers the example embodiment to beginconvolution of the sound, now being output to the headphones. One ormore processors (such as a processor in the PED or in communication withthe PED) convolves the sound so it externally localizes to the listeneras originating at the speaker at the PED. As such, the listener hearsthe sound as originating from the speaker of the PED even though thespeaker is not actually generating the sound anymore. The listenerexperiences little or no change in localization of the sound betweenhearing the sound from the speaker and through the air, to hearing thesound through headphones and convolved to a position at the PED orbehind the sound source or PED.

Consider further this example in which the listener wears headphones andhears binaural sound that originates from the PED. After some time, thevoltage level of the PED drops below a predetermined threshold and apower state of the PED changes to “low battery.” The change in statetriggers the example embodiment to discontinue convolution of the soundto binaural sound and instead to continue to play the sound output tothe headphones in mono sound. Ceasing to execute processes and/orhardware that convolve the sound preserves battery power. This processconservation also improves the performance of the PED by prolonging theduration of the powered-on state of the PED. Further, changing of thesound from binaural sound that externally localizes to the listener tomono sound also improves the experience of the listener by serving as analert to the listener of the “low battery” state of the PED.

FIGS. 5-8 show example embodiments with various electronic devices, suchas a TV or display, HPED, WED, HMD, OHMD, wireless earphones orheadphones. Example embodiments, however, are applied with other typesof electronic devices as well.

FIGS. 5A-5H show a listener 500 interacting with a handheld portableelectronic device (HPED) 510 to change sound between being provided inbinaural sound and being provided in mono or stereo sound. The listenerhears the sound through an electronic device 520, such as headphones,earphones, earbuds, or OHMD. Each figure includes a sound localizationpoint (SLP) 530A-530H shown as an enlarged asterisk (*) that illustratesfrom where the listener hears the sound as originating. Forillustration, the listener 500 is using the HPED 510 to communicate witha user (e.g., talking with a person, an IPA, an IUA, a bot, or acomputer program). For example, the communication occurs during atelephone call or voice exchange with the user. The SLP thus representsan origin of the voice of the user to the listener (e.g., a spot wherethe voice of the user is heard by the listener to originate).

In FIG. 5A, the listener 500 hears the voice of the user in mono orstereo sound through the electronic device 520, and the SLP 530A islocated inside a head of the listener 500. For example, the listener 500receives or initiates a telephone call with the user through the HPED510, and the listener hears the voice of the user in a conventional wayas mono or stereo sound through the electronic device 520.

In FIG. 5B, the listener 500 hears the voice of the user in binauralsound through the electronic device 520, and the SLP 530B is externallylocated outside the head of the listener 500. The HPED 510 displays animage 540 of the user, and the image 540 is the sound source to thelistener. The listener 500 perceives or believes the voice of the useroriginates from the image 540 that is being displayed on the display ofthe HPED. The HPED, however, is not emanating sound through its speakerssince the voice of the user is being provided to the listener 500through the electronic device 520 worn on the head of the listener.

In FIG. 5B, a processor (such as a processor in the HPED 510) convolvesthe voice of the user to a coordinate location 550 that is behind theHPED 510. For illustration, the coordinate location 550 is shown as animage of a head of the user. The coordinate location 550 is located on aline-of-sight 560 that extends from the head or face of the listener500, through the image 540 on the display of the HPED, and to thecoordinate location 550.

When the listener 500 holds the HPED 510 in his or her hand and in frontof his or her face (FIG. 5B), a distance from the head of the listener500 to the HPED 510 is a near-field distance (d1). The coordinatelocation 550, however, is located a distance (d2) away from the HPED 510in order to be at a far-field distance from the listener 500. As such,the distance of the coordinate location 550 from the head of thelistener 500 is greater than or equal to one meter: d1+d2≥1.0 m.

Consider an example in which the HPED 510 is a smartphone, and thelistener 500 in FIG. 5B holds the smartphone one foot in front of hisface (e.g., d1=1.0 ft). The smartphone has a spherical coordinatelocation of (1.0 ft, 0°, 0°) with respect to the head of the listener500 at the origin. In order for the listener to localize a voice of theuser to the smartphone, near-field HRTFs could be selected to havecoordinates that match the spherical coordinate location of thesmartphone (e.g., select an HRTF pair corresponding to a coordinatelocation of (1.0 ft, 0°, 0°). As explained herein, convolving sound withsuch near-field HRTFs can be problematic. Instead of using thesenear-field HRTFs, an example embodiment instead selects far-field HRTFpairs. For example, the smartphone selects far-field HRTFs that have aspherical coordinate location of (3.2 ft, 0°, 0°). The voice of the useris convolved to a location that is behind the smartphone (e.g.,convolved to coordinate location 550). The listener 500 will hear thevoice of the user as if originating from the location of the smartphonesince the coordinate location 550 is aligned with the image 540.

In the example of the HPED being a smartphone, the listener may perceivea discrepancy between the location of the image 540 and the location ofthe coordinate location 550. For instance, the listener localizes thevoice of the user to the coordinate location 550 but sees the image 540of the user at a closer location where the smartphone is located. Thediscrepancy is relatively minor, and further minimized for small valuesof d2. One way to minimize or eliminate the discrepancy is to place thecoordinate location 550 on the line-of-sight 560. Convolving the voicealong the line-of-sight reduces or eliminates a discrepancy inlocalization between the percept of the image 540 at a distance of d1and a binaural sound convolved to a different coordinate location 550.

FIGS. 5C and 5D show examples of how the listener 500 easily and quicklyswitches between hearing the sound as binaural sound and hearing thesound as mono or stereo sound.

In FIG. 5C, the listener 500 flicks or rotates the HPED 510 from a firstposition (shown in dashed lines with a display facing toward the faceand/or head of the listener) to a second position (shown in solid lineswith a display facing the ground or away from the face and/or head ofthe listener). The rotation is shown with arrow 560. After the HPED 510rotates per arrow 560, the sound localizes to the listener as mono orstereo sound inside his or her head at the SLP 530C. When the HPEDdetects or senses the rotation, sound changes from being provided asbinaural sound to being provided as mono or stereo sound.

In FIG. 5D, the listener 500 flicks or rotates the HPED 510 from a firstposition (shown in dashed lines as facing the ground or away from theface and/or head of the listener) to a second position (shown in solidlines as facing the face and/or head of the listener). The rotation isshown with arrow 570. After the HPED 510 rotates per arrow 570, thelistener localizes the sound as binaural sound outside his or her headat the SLP 530D. When the HPED detects or senses the rotation, soundchanges from being provided as mono or stereo sound to being provided asbinaural sound.

FIGS. 5E-5G show more examples how the listener 500 easily and quicklyswitches between hearing the sound as binaural sound and hearing thesound as mono or stereo sound.

In FIG. 5E, the facing sensor or camera 580 captures an image of theface of the listener 500. The listener continues to localize thebinaural sound externally to the HPED at SLP 530E as long as the face ofthe listener 500 remains in a field-of-view (FOV) 582 of the camera.FIG. 5E shows the listener rotating the HPED 510 by an azimuth angle(θ). As the listener rotates the HPED within the FOV 582, he or shecontinues to localize the sound to the SLP 530E while the soundcontinues to be convolved to the coordinate location 550 that is behindthe HPED.

FIG. 5F shows the listener 500 rotating the HPED 510 so the face of thelistener is outside the FOV 582 of the facing sensor or camera 580. Themovement causes the sound to change from being provided to the listeneras binaural sound to being provided to the listener as mono or stereosound. When the HPED detects or senses that the face of the listener isno longer in the FOV of the camera, sound changes from being provided asbinaural sound to being provided as mono or stereo sound. The listenernow localizes the sound as mono or stereo sound to the SLP 530F locatedinside the head of the listener.

In FIG. 5G, the sound switches between binaural sound and mono or stereosound when the HPED is placed onto or removed from an object 590 (suchas a table or other surface). For example, when the HPED 510 is placedon the object 590, the display, camera, or other sensor is covered,obstructed, or darkened. Sensing the placement or the action of theplacement triggers the processor to change how the sound is convolvedand/or how the sound is provided to the listener. For instance, inresponse to sensing the HPED being placed on the object 590, change thesound from being provided to the listener as binaural sound to beingprovided as mono or stereo sound. The listener now localizes the mono orstereo sound to the SLP 530G located inside the head of the listener.When the listener retrieves or removes the HPED from the object 590, thesound changes from being provided as mono or stereo sound to beingprovided as binaural sound that externally localizes to the listener.

FIG. 5H shows that the coordinate location 550 remains behind the HPED510 and on the line-of-sight 560 as the listener 500 moves the HPEDwhile communicating with the user. For example, the listener moves theHPED from the position shown in FIG. 5B to the position shown in FIG.5H. The origination point of the voice of the user thus tracks theposition of the HPED as the listener holds the HPED in his or her handand moves the HPED to different locations. The listener hears the voiceof the user as binaural sound at or near 530H as if the voice wereoriginating from or emanating from the location of the HPED when in factthe listener hears the voice of the user through the electronic device520.

As the listener moves the HPED during the communication with respect tohis or her face or head orientation, the sound localization information(SLI) processing the voice of the user changes. For example, theprocessor convolves or processes the voice of the user with new ordifferent sets of HRTFs in order to change the coordinate location 550so that it remains behind the HPED relative to the user during thecommunication.

SLPs 530B, 530E, and 530H appear at, slightly behind, and slightly infront of the sound source at the HPED 510. This illustrates that alistener may not localize a SLP to an exact coordinate location. Alistener associates the sound convolved to location 550 with the visibleHPED 510 and the association results in the listener localizing theconvolved sound at the area occupied by the sound source. The SLP maynot have a precise coordinate location, the SLP can vary depending onthe circumstances such as the type of sound, the auditory environment,the sensitivity of the listener, the visibility of the sound source, adistance from the listener to the sound source, and other factors.

An example embodiment determines a gaze direction or looking directionof the listener independent of the orientation of the head of thelistener. For example, a gaze tracker or detector, facing sensor orother sensor monitors the gaze angle of the listener, or a cameracaptures an image of the face of the listener and the image is analyzedto determine the direction that the listener is looking. The exampleembodiment selects HRTF pairs for coordinates that intersect or coincidewith a line extending from the head of the listener in the direction ofthe gaze or the looking direction. The listener hears the soundconvolved such that the sound appears to originate from the directionthat the listener is looking.

Consider an example where the sound is convolved to a point on the otherside of a sound source (such as a HPED and/or other objects) and doesnot localize to the listener unless the gaze angle of the listenerintersects with the sound source. The listener hears the soundexternally localized when the listener looks at the sound source, suchas the HPED or other object. When the gaze angle is not directed to thesound source then the sound of the sound source is paused, muted, orplayed without being externally localized (e.g., the sound is providedto the listener as mono sound or stereo sound).

Consider a similar example in which the listener hears localizations ofbinaural sounds at or beyond one or more sound sources while the gaze ofthe listener is not toward the sound source. When the gaze is detectedas being toward a sound source, the sound of the sound source isswitched to mono sound or stereo sound, paused, muted, or adjusted inanother way. This allows the listener to focus on a particular soundsource by looking at the sound source. While looking at the soundsource, the sound source localizes in his or her head, while other soundsources continue to externalize out and away from the listener.

Consider an example in which the listener holds the HPED in front of hisor her face during the communication at spherical coordinate location(r, θ, ϕ) of (0.3 m, 0°, 0°). A processor processes the voice with HRTFscorresponding to coordinates (1.0 m, 0°, 0°) so the voice of the userlocalizes for the listener on the far side of the HPED from thelistener. The listener then rotates his or her head on the vertical orlongitudinal axis by negative twenty degrees (θ=−20°). In response tothe change in head orientation, the processor convolving the voice ofthe user retrieves a new HRTF pair corresponding to coordinates (1.0 m,+20°, 0°) and processes the voice of the user with the new HRTF pair.The voice of the user continues to localize to the listener as if fromthe HPED.

Consider an example in which a person holds a HPED in his or her handand is on a telephone call with another user. The HPED includes a camerathat captures an image of the person and also includes a display thatdisplays an image of the user as the sound source. A facial recognitionand/or head tracking system monitor a head orientation of the personduring the telephone call. When the person changes his or her headorientation by or beyond a predetermined amount or threshold amount, thevoice of the user that the person hears changes from binaural sound tomono sound or from mono to binaural. For example, the HPED changes thevoice of the user from binaural sound to sound that the person localizesinside his or her head in response to the HPED detecting a predeterminedchange in the head orientation. For example, the change of theorientation of the head is relative to the HPED such as an azimuth anglechange due to head rotation on a vertical axis of the head and/or achange in the pitch angle of the head resulting from head rotation onthe frontal axis of the head. For instance, the predetermined amount orthreshold includes rotation in one or more axes by one or more oftwenty-five degrees (25°), thirty degrees (30°), thirty-five degrees(35°), forty degrees (40°), forty-five degrees (45°), fifty degrees(50°), fifty-five degrees (55°), sixty degrees (60°), sixty-five degrees(65°), seventy degrees (70°), seventy-five degrees (75°), eighty degrees(80°), eighty-five degrees (85°), or ninety degrees (90°). Thesepredetermined amounts or thresholds are applied to other exampleembodiments as well.

FIGS. 6A and 6B show an electronic or computer system 600 in which alistener 610 hears voices from three different users whose voiceslocalize externally to the listener 610 as binaural sound to threedifferent images 620A, 620B, and 620C on a display 630 of an electronicdevice 640. The listener wears an electronic device 612 (e.g., wirelessearphones, wireless headphones, a HMD, or OHMD) that wirelesslycommunicate with the electronic device 640 to hear the voices asbinaural sound. By way of example, the three different users are peopleengaged in a telephone call with the listener 610, images displayedduring an advertisement, or computer programs (e.g., an IPA, IUA, bot,characters in a computer game, independent processes or tasks, windows,etc.). The electronic device 640 is shown as a flat display but includesdifferent types of displays and electronic devices, such as a television(TV), a curved display, an edge display, a 3D TV, a projector andprojection, a virtual display, etc.

The electronic device 640 includes a sensor 650 that determines one ormore of a presence or existence of a person or avatar, a distance to aperson or avatar, a gaze or looking direction of a person or avatar, avoice of a person or avatar, a facing direction of a person or avatar,and gestures of a person or avatar. For example, the sensor 650 is amotion sensor, presence sensor, camera, proximity sensor, infraredsensor, facing sensor, a virtual sensor, a position of a VR camera orvirtual point-of-view (POV), etc.

The listener 610 sees each image 620A, 620B, and 620C along a respectiveline-of-sight 660A, 660B, and 660C while the listener stands anear-field distance (d) from the display 630. The listener, however,perceives the images at different distances with respect to thelistener. Images 620A and 620B appear to be located at the surface ofthe display 630. Image 620C, however, appears to be located farther awaythan a wall or virtual wall 622 at which the display 630 is located. Forexample, the perception of the greater distance to image 620C is createdby the display 630 being a display that provides for the viewer aperception of distance or 3D space (e.g., a 3D TV or display, a HMD, adisplay showing stereoscopic video or images). A distance (d1) from thelistener 610 to the location of the image 620A is a near-field distance.A distance (d2) from the listener 610 to the location of the image 620Bis a near-field distance. A distance (d3) from the listener 610 to thelocation of the image 620C is a far-field distance.

Since distances d1 and d2 are near-field distances, voices of the userscorresponding to images 620A and 620B are convolved with HRTFs havingcoordinate locations 670A and 670B that are far-field distances. Sincedistance d3 is a far-field distance, the voice of the user correspondingto image 620C is convolved with a HRTF pair having a distance coordinatethat matches or corresponds to the far-field distance of d3. Thelistener perceives a voice at image 620A at its location on the display630 even though the voice is convolved with HRTFs of coordinate location670A. The listener perceives a voice at image 620B at its location onthe display 630 even though the voice is convolved with HRTFs ofcoordinate location 670B. The listener perceives a voice at image 620Cat its location behind or beyond the display 630 where the listenervisually localizes the image 620C and where the sound is convolved tooriginate. Coordinate locations 670A and 670B are behind wall 622.

FIG. 7 shows an electronic or computer system 700 in which two listeners710 and 720 are engaged in a telephone call in accordance with anexample embodiment.

Listener 710 sits in a chair 711 with a laptop computer 722 resting on atable 724 while the listener 710 talks to listener 720. An image 730 ofthe listener 720 appears on the display of the laptop computer 722. Thelistener 710 hears a voice of the listener 720 through headphones orearphones 732 that wirelessly communicate with the laptop computer 722.

A voice of the listener 720 localizes to the listener 710 as binauralsound that is heard to emanate from or originate from the image 730 onthe display of the laptop computer 722. A processor (e.g., a processorin the laptop computer 722 or elsewhere in the computer system 700)processes the voice of the listener 720 with HRTFs for coordinatelocation 736 that is behind the laptop computer 722.

The coordinate location 736 is on a line-of-sight 738 that extends froma head of the listener 710 to the image 730 displayed on the laptopcomputer 722. The coordinate location 736 is shown in empty space behindthe laptop computer 722. A distance (d1) from the head of the listener710 to the image 730 is a near-field distance that is less than onemeter. A distance (d1+d2) from the head of the listener 710 to thecoordinate location 736 is a far-field distance that is one meter ormore. As such, the processor processes or convolves the voice of thelistener 720 with far-field HRTFs even though the image 730 where thelistener perceives the origination of the voice is located at anear-field distance with respect to the listener 710.

Listener 720 talks to listener 710 while listener 720 sits in a chair721 with a smartphone 752 in a hand of the listener in front of his orher face. An image 760 of the listener 710 appears on the display of thesmartphone 752. The listener 720 hears a voice of the listener 710through headphones or earphones 754 that wirelessly communicate with thesmartphone 752. The smartphone 752 wirelessly communicates with thelaptop computer 722 over one or more networks 760 while the listeners710 and 720 are remote from each other at different geographicallocations.

A voice of the listener 710 localizes to the listener 720 as binauralsound perceived by listener 720 as emanating from or originating fromthe image 760 on the display of the smartphone 752. A processor (e.g., aprocessor in the smartphone 752 or elsewhere in the computer system 700)processes the voice of the listener 710 with HRTFs having a coordinatelocation 766 that is behind the smartphone 752. The coordinate location766 is located on a floor 770 located next to a stack of books 772 in aroom where the listener 720 is located.

The coordinate location 766 is on a line-of-sight 768 that extends fromthe listener 720 to the image 760 displayed on the smartphone 752. Adistance (d3) from a head of the listener 720 to the image 760 is anear-field distance that is less than one meter. A distance (d3+d4) fromthe head of the listener 720 to the coordinate location 766 is afar-field distance that is one meter or greater. As such, the processorprocesses or convolves the voice of the listener 710 with far-fieldHRTFs even though the image 760 where the voice is perceived is locatedat a near-field distance with respect to the listener 720.

FIGS. 8A and 8B show an electronic or computer system 800 in which alistener 810 sits at a table 820 and engages in a telephone call orother type of communication with two users via a portable electronicdevice 840. Images 830A and 830B of the two users appear on a display850 of the portable electronic device 840 while the listener 810 talksto the users. The images 830A and 830B of the two users move andinteract with the listener 810 in real-time. For example, the images aresoftware programs or people with whom the listener 810 talks. Forinstance, the users are people and the images 830A and 830B arereal-time video of the users during a conference call. As anotherexample, one user is a person and one user is a bot; the images of bothusers are avatars; and the avatars are visually perceived by listener810 as two-dimensional or three-dimensional AR images in a 3Denvironment of the listener (e.g., using stereoscopic video, holograms,light fields, or another type of display or projection).

The display 850 is divided into two halves or two sections 855A and8558. Section 855A displays image 830A, and section 855B displays image830B. This situation occurs, for example, when the listener 810 talkswith two different users. Example embodiments include a listenerengaging in visual telephony with a different number of other users,such as a single user, three users, ten users, twenty users, or anothernumber of users. Each user may be presented as an image and/or a sectionor area of a display or other location (including AR and VR images).

A camera 860 in the portable electronic device 840 captures an image ofthe listener 810 and provides the image and/or video to one or more ofthe users. Further, the camera 860 and one or more software programsperform other functions, such as detect and recognize a face of thelistener 810, determine or monitor a presence of the listener 810,determine a distance of the face of the listener from the display 850,determine an angle of the face of the listener or an orientation of thehead of the listener 810 relative to the camera 860 and/or relative tothe display 850 of the electronic device 840, and perform otherfunctions discussed herein.

The listener 810 is located a near-field distance from the display 850of the electronic device 840. HRTFs having near-field distancecoordinates may not be available or desirable for convolution of thevoices of the users for the listener in a near-field range. Instead, anexample embodiment convolves the voices with HRTFs having far-fielddistance coordinates. The coordinate locations of these HRTFs fallbeyond or behind the display 850 of the electronic device 840.

A processor processes or convolves the voice of each user with adifferent pair of HRTFs so the voices are not heard to overlap orlocalize from a coincident or matching location during thecommunication. For example, the processor convolves the voicecorresponding to image 830A with HRTFs having a coordinate location at870A and convolves the voice of image 830B with HRTFs having acoordinate location at 870B. These locations are separated from eachother on opposite sides of azimuth angle A3 having a vertex at the headof the listener. Coordinate locations 870A and 870B are shown to existin empty space behind the display 850 of the electronic device 840 withrespect to the location of the head of the listener 810.

The listener 810 hears a voice from image 830A as binaural sound thatoriginates from a location 875A. The location 875A corresponds to ormatches the location of the image 830A on the display 850 relative tothe orientation of the head of the listener. The listener 810 hears avoice at image 830B as binaural sound originating from location 875B.Location 875B corresponds to or matches the position of the image 830Bon the display 850 as heard by the listener 810.

The listener 810 has a forward-looking direction off to the side of thedisplay 850 with a line-of-sight 880 along θ=0°. The voice of a user isconvolved with HRTFs included within a range of coordinate locationsthat provide for the binaural sound to be rendered behind the respectiveimage of the user (here image 830A or 830B) and behind the display 850relative to the head of the listener 810.

Rays R1-R4 extending from the head of the listener 810 and angles A1-A3having vertices at the head of the listener 810 are shown to illustrateexample ranges for azimuth coordinates of HRTF pairs. The voice or othersound of a user is convolved with an HRTF pair having an azimuthcoordinate within the range of the angle occupied by the image ordisplay section corresponding to the user. Convolving the voice with anHRTF pair within the azimuth range allows the listener 810 to externallylocalize the voice or other sound of a user such that the listenerexperiences the binaural sound as originating at the image of the user.Ray R1 has an azimuth coordinate of θ1. Ray R2 has an azimuth coordinateof θ2. Ray R3 has an azimuth coordinate of θ3. Ray R4 has an azimuthcoordinate of θ4. Angle A1 has sides R3 and R4. Angle A2 has sides R1and R2. Angle A3 has sides R1 and R4. As such the size of angle A1 isthe difference of θ4 and θ3, the size of angle A2 is the difference ofθ2 and θ1, and the size of angle A3 is the difference of θ4 and θ1.

To localize the sound corresponding to image 830A, an example embodimentselects HRTF pairs having azimuth angle coordinates between θ3 and θ4.Here, θ4 is greater than θ3 and less than or equal to the azimuthcoordinate of location 870A. The azimuth angle of ray R4 and ofcoordinate location 870A represents the alignment of a right-side edgeor boundary of image 830A from the POV of the head of the listener 810that has a forward-facing orientation of θ=0°. The azimuth angle of rayR3 represents the alignment of a left-side edge or boundary of image830A with respect to the head of the listener. The listener 810 observesthe image 830A within the angle A1. The listener 810 hears the sound ofthe voice of a user originating from within the angle A1 and from theimage 830A when the voice is convolved with a HRTF pair having anazimuth coordinate θ such that θ3≤θ≤θ4. For example, location 870A hasan azimuth coordinate equal to θ4 and so the voice of the user shown as830A is convolved with a HRTF pair corresponding to location 870A, andlocalizes to the listener at image 830A (e.g., 875A).

Sound is convolved behind image 830B by HRTFs with an azimuth angle inthe range between θ1 and θ2 or interior to the angle A2. Ray R1represents an edge or boundary for where the image 830B is seen by thelistener from the head orientation of forward facing ray 880. Anopposite edge or boundary of the image 830B as seen by the listener withthe head oriented toward θ=0° is represented by ray R2. The azimuthangle of coordinate location 870B is a value between or included by theazimuth angle coordinate of ray R1 and ray R2. As such, the listener 810perceives sound convolved to location 870B as emanating from the image830B such as from point 8758.

The images 830A and 830B of the users appear close beside each other onthe display 850. The voices of the users, however, are convolved todistant points 870A and 870B on opposite sides of angle A3 that arefarther apart than the images of the users. This improves the userexperience by preventing the listener from localizing the two voices asoverlapping and assists the listener in spatially distinguishing the twosound sources from each other.

For ease of illustration azimuth angles of FIG. 8A are shown anddiscussed herein. Example embodiments also similarly calculate elevationcoordinates in order to select HRTF pairs that are aligned with theimage or sound source from the point of view of the head of thelistener. The example embodiments select elevation angles from thosethat fall between a ray that bounds a lower edge of an image, area of adisplay, or sound source and a ray that bounds an upper edge of animage, area of display, or sound source (the rays having endpoints atthe head of the listener).

Providing HRTFs with a range of different azimuth (and/or elevation)angles solves a technical problem and assists in convolving the voicesof the users. For example, HRTFs corresponding to coordinate location870A may not be available. In this situation, other HRTFs are selectedthat correspond to coordinate locations with an azimuth angle interiorof angle A1 (i.e., behind the display). Likewise, HRTFs havingcoordinate locations corresponding to the location of 870B may not beavailable. In this situation, HRTFs with other coordinate locationscould be selected as long as the selected coordinate locations arealigned with the image 830B (e.g., with an azimuth angle within angleA2). Thus the objective of assisting the listener to localize a voice ofa user at the image or sound source representing the user can beachieved with multiple different HRTF pairs.

In FIG. 8B, the listener 810 has changed the position of the portableelectronic device (PED) 840 to be directly in front of the listener. TheHRTF pairs convolving the sound of the users have been updated tocompensate for the change in the distance to and orientation of the headof the listener 810 relative to the images of the users displayed on thedisplay 850. For example, the changes in the relative position of thehead are determined by analyzing the facial profile angle from one ormore images or video captured by the camera 860, or by resolving headmovement reported by a head tracking system with the movement of the PED840 as reported by sensors of the PED. The user being represented byimage 830B terminates or ceases the communication with the listener 810.The image 830B is replaced with an image 890 of the listener 810. Thecamera 860 captures an image of the listener 810, and the electronicdevice 840 displays the image 890 on the display 850. During thecommunication, the listener 810 simultaneously sees a real-time image890 of himself or herself and a real-time image 830A of the other user.

FIG. 9 is a method that selects HRTFs based on a distance of a listenerfrom a sound source in accordance with an example embodiment.

Block 900 states determine a distance from a listener to a sound source.

One or more electronic devices or sensors of an electronic devicedetermine a distance from the listener to the sound source. By way ofexample, the sensors include one or more of a camera, a proximitysensor, ultrasonic sensor, a radio frequency identification (RFID) tagand reader, laser, light source sensor, or other sensor that assists indetermining a distance of an object from an electronic device.

The distance can represent the distance from the listener to anelectronic device (e.g., an electronic device representing the soundsource), to a display (e.g., a flat or curved display displaying thesound source), to an AR image rendering to a listener as though in thephysical environment, to a VR image in a virtual environment, or toanother sound source.

As an example, a camera at the location of the sound source captures animage of a face of a listener and executes an algorithm to determine adistance of the face from the camera. For example, triangle similaritycan be used to determine distance from the camera to a known object(e.g., a face) based on a size of the head and/or face of the capturedimage. For instance, a distance (D) to the face is a function of a knownwidth of the face (WF) times the focal length (FL) of the camera dividedby the width of the pixels (WP). As another example, a facialrecognition can be used to determine distance from the camera to theknown object (e.g., a face) based on distances between one or morefacial landmarks (e.g., eyes, nose, mouth, etc.).

As another example, a camera that is in communication with theelectronic system captures an image that includes both the listener andthe sound source that are away from the camera. The electronic systemthen uses the image to determine the life-scale distance between thelistener and sound source included in the image. For example, imagerecognition software determines an angle and distance from the camera tothe listener (side A of a triangle) and an angle and distance from thecamera to the sound source (side B of the triangle). An algorithmexecuting in the electronic system sums the two angles to determine thevertex angle at the camera between the listener and the sound source(i.e., between side A and side B). The algorithm then uses the law ofcosines to calculate the distance between the listener and the soundsource (i.e., side C of the triangle).

As another example, the listener wears a HPED (e.g., a smartphone) orwearable electronic device (e.g., a HMD) that displays an AR or VR imageof the sound source rendered to be perceived at a particular distancewith respect to an origin, such as a location of the wearer. A distancefrom the listener to the sound source is based on a relative size of theimage being displayed or a location or virtual location to where theimage is being rendered. For instance, the image is displayed at a realor virtual chair that is observed as two feet away from the wearer ofthe electronic device.

As another example, the listener wears electronic earphones orheadphones that communicate with the HPED or other electronic devicethat displays the sound source. Communication between these twoelectronic devices establishes a distance from the listener wearing theelectronic earphones or headphones and a sound source at the HPED orother electronic device.

As another example, software and/or hardware that positions, displays,and/or monitors locations of virtual objects in VR or AR is queried forand/or reports the distance between a head or virtual head of thelistener and a sound source.

Block 910 makes a determination as to whether the distance is anear-field distance.

A near-field distance is a distance less than one meter. A far-fielddistance is a distance greater than or equal to one meter.

If the answer to the determination is “no” then flow proceeds to block920 that states process and/or convolve the sound with HRTFs havingfar-field coordinates that correspond to or match with the coordinatelocation of the sound source. For example, the determined distance fromthe listener to the sound source matches or equals a distance from thelistener to the coordinate location corresponding to the HRTF pair beingexecuted to convolve the sound.

A processor (such as a digital signal processor or DSP) processes thesound with SLI (including a pair of HRTFs). The HRTFs have a coordinatelocation that matches, approximates, or corresponds to a location of thesound source.

If the answer to the determination is “yes” then flow proceeds to block930 that states process and/or convolve the sound with HRTFs havingfar-field coordinates so the sound localizes to the listener at thesound source.

A processor (such as a digital signal processor or DSP) processes thesound with SLI (including a pair of HRTFs). The HRTFs have a coordinatelocation that does not match, approximate, or correspond to a locationof the sound source. For example, the HRTFs have a coordinate locationthat is away from or farther than the location of the sound source. Forinstance, the HRTFs have spherical coordinates with a distance (r)coordinate that is larger than or greater than the distance from thelistener to the sound source.

An example embodiment executes one or more of the following so the soundlocalizes to the listener at the sound source when the HRTFs have acoordinate location that does not match the location of the soundsource: convolve the sound to a location on, near, along, or about theline-of-sight from the listener to the sound source, convolve the soundto a location that is behind or beyond the sound source, convolve thesound to a location that is behind or beyond the electronic deviceproviding or displaying the sound source (e.g., an image of the soundsource), and convolve the sound to have a volume or loudnesscommensurate with the distance from the listener to the sound source (asopposed to the distance from the listener to the coordinate location ofthe HRTFs).

By way of example, the sound localization information (SLI) areretrieved, obtained, or received from memory, a database, a file, anelectronic device (such as a server, cloud-based storage, or anotherelectronic device in the computer system or in communication with a PEDproviding the sound to the user through one or more networks), etc. Forinstance, the information includes one or more of HRTFs, ILDs, ITDs,and/or other information discussed herein. Instead of being retrievedfrom memory, this information can also be calculated in real-time.

An example embodiment processes and/or convolves sound with the SLI sothe sound localizes to a particular area or point with respect to auser. The SLI required to process and/or convolve the sound is retrievedor determined based on a location of a desired SLP or the sound source.For example, if the SLP is to be located one meter in front of a face ofthe listener and slightly off to a right side of the listener, then anexample embodiment retrieves the corresponding HRTFs, ITDs, and ILDs andconvolves the sound to this location. The location can be more specific,such as a precise spherical coordinate location of (1.2 m, 25°, 15°),and the HRTFs, ITDs, and ILDs are retrieved that correspond to thelocation. For instance, the retrieved HRTFs have a coordinate locationthat matches or approximates the coordinate location where sound isdesired to originate to the user. Alternatively, the location is notprovided but the SLI is provided (e.g., a software application providesto the DSP HRTFs and other information to convolve the sound).

A central processing unit (CPU), processor (such as a DSP), ormicroprocessor processes and/or convolves the sound with the SLI, suchas a pair of head related transfer functions (HRTFs), ITDs, and/or ILDsso that the sound will localize to a zone or SLP. For example, the soundlocalizes to a specific point (e.g., localizing to point (r, θ, ϕ)) or ageneral location or area (e.g., localizing to far-field location (θ, ϕ)or near-field location (θ, ϕ)). As an example, a lookup table thatstores a set of HRTF pairs includes a field/column that specifies thecoordinates associated with each pair, and the coordinates indicate thelocation for the origination of the sound. These coordinates include adistance (r) or near-field or far-field designation, an azimuth angle(θ), and/or an elevation angle (ϕ).

The complex and unique shape of the human pinnae transforms sound wavesthrough spectral modifications as the sound waves enter the ear. Thesespectral modifications are a function of the position of the source ofsound with respect to the ears along with the physical shape of thepinnae that together cause a unique set of modifications to the soundcalled head related transfer functions or HRTFs. A unique pair of HRTFs(one for the left ear and one for the right ear) can be modeled ormeasured for each position of the source of sound with respect to alistener.

A HRTF is a function of frequency (f) and three spatial variables, byway of example (r, θ, ϕ) in a spherical coordinate system. Here, r isthe radial distance from a recording point where the sound is recordedor a distance from a listening point where the sound is heard to anorigination or generation point of the sound; δ (theta) is the azimuthangle between a forward-facing user at the recording or listening pointand the direction of the origination or generation point of the soundrelative to the user; and ϕ (phi) is the polar angle, elevation, orelevation angle between a forward-facing user at the recording orlistening point and the direction of the origination or generation pointof the sound relative to the user. By way of example, the value of (r)can be a distance (such as a numeric value) from an origin of sound to arecording point (e.g., when the sound is recorded with microphones) or adistance from a SLP to a head of a listener (e.g., when the sound isgenerated with a computer program or otherwise provided to a listener).

When the distance (r) is greater than or equal to about one meter (1 m)as measured from the capture point (e.g., the head of the person) to theorigination point of a sound, the sound attenuates inversely with thedistance. One meter or thereabout defines a practical boundary betweennear-field and far-field distances and corresponding HRTFs. A“near-field” distance is one measured at about one meter or less;whereas a “far-field” distance is one measured at about one meter ormore. Example embodiments are implemented with near-field and far-fielddistances.

The coordinates for external sound localization can be calculated orestimated from an interaural time difference (ITD) of the sound betweentwo ears. ITD is related to the azimuth angle according to, for example,the Woodworth model that provides a frequency independent ray tracingmethodology. The coordinates (r, θ, ϕ) for external sound localizationcan also be calculated from a measurement of an orientation of and adistance to the face of the person when a head related impulse response(HRIR) is captured.

The coordinates can also be calculated or extracted from one or moreHRTF data files, for example by parsing known HRTF file formats, and/orHRTF file information. For example, HRTF data is stored as a set ofangles that are provided in a file or header of a file (or in anotherpredetermined or known location of a file or computer readable medium).The data can include one or more of time domain impulse responses (FIRfilter coefficients), filter feedback coefficients, and an ITD value.This information can also be referred to as “a” and “b” coefficients. Byway of example, these coefficients are stored or ordered according tolowest azimuth to highest azimuth for different elevation angles. TheHRTF file can also include other information, such as the sampling rate,the number of elevation angles, the number of HRTFs stored, ITDs, a listof the elevation and azimuth angles, a unique identification for theHRTF pair, and other information. The data can be arranged according toone or more standard or proprietary file formats, such as AES69, andextracted from the file.

The coordinates and other HRTF information are calculated or extractedfrom the HRTF data files. A unique set of HRTF information (including r,θ, ϕ) is determined for each unique HRTF.

The coordinates and other HRTF information are also stored in andretrieved from memory, such as storing the information in a look-uptable. The information is quickly retrieved to enable real-timeprocessing and convolving of sound using HRTFs and hence improvescomputer performance of execution of binaural sound.

The SLP represents a location where a person will perceive an origin ofthe sound. For an external localization, the SLP is away from the person(e.g., the SLP is away from but proximate to the person or away from butnot proximate to the person). The SLP can also be located inside thehead of the person (e.g., when the sound is provided as mono sound orstereo sound).

A location of the SLP corresponds to the coordinates of one or morepairs of HRTFs, or corresponds to a coordinate location or zone where alistener perceives a localization due to the influence of a visual cue(e.g., an image, object, or device), or a known or understood locationof a sound source that is not visible (e.g., a recently witnessed oranticipated location of a sound source, a device that is out of afield-of-view, an avatar suddenly occluded by another virtual object).

For example, the coordinates of or within a SLP or a zone match orapproximate the coordinates of a HRTF. Consider an example in which thecoordinates for a pair of HRTFs are (r, θ, ϕ) and are provided as (1.2meters, 35°, 10°). A corresponding SLP or zone intended for a personthus includes (r, θ, ϕ), provided as (1.2 meters, 35°, 10°). In otherwords, the person will localize the sound as occurring 1.2 meters fromhis or her face at an azimuth angle of 35° and at an elevation angle of10° taken with respect to a forward-looking direction of the person. Inthe example, the coordinates of the SLP and HRTF match.

As another example, a listener perceives a SLP at a sound source of anear-field physical object and the HRTFs convolving the perceived soundhave far-field coordinates along the line-of-sight from the head of thelistener to the object. In this case, one or more of the SLP coordinatesdo not match the coordinates of the object (e.g., the r coordinates donot match). HRTF pairs having coordinates matching the coordinates ofthe object may be unavailable to the listener. Later as the listenerlocalizes the sound to the object, a barrier is placed such that thelistener is blocked from seeing the object. The listener continues tolocalize the sound to the object, with the object as the SLP because thelistener remains aware that the object has not moved.

The listener turns around 180° so that the sound source is behind thelistener, and an example embodiment updates the azimuth coordinate ofthe HRTFs by 180°. Though the object is outside of the field of view ofthe listener, the listener continues to localize the sound to thelocation of the object that is behind the listener. The listener havingwitnessed and associated the sound with the object remains aware of thesource of the sound and continues to localize the sound to the locationor area of the object.

SLI can also be approximated or interpolated based on known data orknown SLI such as SLI for other coordinate locations. For example, a SLPis desired to localize at coordinate location (2.0 m, 0°, 40°), butHRTFs for the location are not known. HRTFs are known for twoneighboring locations, such as known for (2.0 m, 0°, 35°) and (2.0 m,0°, 45°), and the HRTFs for the desired location of (2.0 m, 0°, 40°) areapproximated from the two known locations. These approximated HRTFs areprovided to convolve sound to localize at the desired coordinatelocation (2.0 m, 0°, 40°).

Sound is convolved either directly in the time domain with a finiteimpulse response (FIR) filter or with a Fast Fourier Transform (FFT).For example, an electronic device convolves the sound to one or moreSLPs using a set of HRTFs, HRIRs, BRIRs, or RIRs and provides the personwith binaural sound.

In an example embodiment, convolution involves an audio input signal andone or more impulse responses of a sound originating from variouspositions with respect to the listener. The input signal is a limitedlength audio signal (such as a pre-recorded digital audio file) or anongoing audio signal (such as sound from a microphone or streaming audioover the Internet from a continuous source). The impulse responses are aset of HRIRs, BRIRs, RIRs, etc.

Convolution applies one or more FIR filters to the input signals andconvolves the input signals into binaural audio output or binauralstereo tracks. For example the input signals are convolved into binauralaudio output that is specific or individualized for the listener basedon one or more of the impulse responses to the listener.

The FIR filters are derived binaural impulse responses that are executedwith example embodiments discussed herein (e.g., derived from signalsreceived through microphones placed in, at, or near the left and rightear channel entrance of the person). Alternatively or additionally, theFIR filters are obtained from another source, such as generated from acomputer simulation or estimation, generated from a dummy head,retrieved from storage, etc. Further, convolution of an input signalinto binaural output can include sound with one or more ofreverberation, single echoes, frequency coloring, and spatialimpression.

Processing of the sound also includes calculating and/or adjusting aninteraural time difference (ITD), an interaural level difference (ILD),and/or other aspects of the sound in order to alter the cues andartificially alter the point of localization. Consider an example inwhich the ITD is calculated for a location (θ, ϕ) with discrete Fouriertransforms (DFTs) calculated for the left and right ears. The ITD islocated at the point for which the function attains its maximum value,known as the argument of the maximum or arg max as follows:

${I\; T\; D} = {\arg\;{\max(\tau)}{\sum\limits_{n}{{d_{l,\theta,\phi}(n)} \cdot {{d_{r,\theta,\phi}\left( {n + \tau} \right)}.}}}}$

Subsequent sounds are filtered with the left HRTF, right HRTF, and/orITD so that the sound localizes at (r, θ, ϕ). Such sounds includefiltering stereo and monaural sound to localize at (r, θ, ϕ). Forexample, given an input signal as a monaural sound signal s(n), thissound is convolved to appear at (θ, ϕ) when the left ear is presentedwith:s _(l)(n)=s(n−ITD)·d _(l,θ,ϕ))(n);and the right ear is presented with:s _(r)(n)=s(n)·d _(r,θ,ϕ)(n).

Consider an example in which a dedicated digital signal processor (DSP)executes frequency domain processing to generate real-time convolutionof monophonic sound to binaural sound.

By way of example, a continuous audio input signal x(t) is convolvedwith a linear filter of an impulse response h(t) to generate an outputsignal y(t) as follows:

y(τ) = x(τ) ⋅ h(τ) = ∫₀^(∞) x(τ − t) ⋅ h(t) ⋅ d t.

This reduces to a summation when the impulse response has a given lengthN and the input signal and the impulse response are sampled at t=iDt asfollows:

${y(i)} = {\underset{j = 0}{\sum\limits^{N - 1}}{{x\left( {i - j} \right)} \cdot {{h(j)}.}}}$

Execution time of convolution further reduces with a Fast FourierTransform (FFT) algorithm and/or Inverse Fast Fourier Transform (IFFT)algorithm.

Consider another example of binaural synthesis in which recorded orsynthesized sound is filtered with a binaural impulse response (e.g.,HRIR or BRIR) to generate a binaural output sound to the person. Theinput sound is preprocessed to generate left and right audio streamsthat are mapped to one or more sound sources or sound localizationpoints (known as SLPs). These streams are convolved with a binauralimpulse response for the left ear and the right ear to generate the leftand right binaural output sound signal. The output sound signal isfurther processed depending on a final destination. For example across-talk cancellation algorithm is applied to the output sound signalwhen it will be provided through loudspeakers or applying artificialbinaural reverberation to provide 3D spatial context to the sound.

Example embodiments designate or include an object, sound source, image,or device on the ray that extends from a head of a listener to theintended SLP (such as displaying an image as the sound source at or inline with the intended SLP). For an external localization, the SLP isaway from the person (e.g., the SLP is away from but proximate to theperson or away from but not proximate to the person). The SLP can alsobe located inside the head of the person (e.g., when sound is providedto the listener in stereo or mono sound).

Block 940 states provide the sound to the listener as binaural soundthat localizes to the listener at the sound source.

Binaural sound is provided to the listener through one or moreelectronic devices including, but not limited to, one or more of boneconduction headphones, speakers of a wearable electronic device (e.g.,headphones, earphones, electronic glasses, earbuds, head mounteddisplay, smartphone, etc.). Binaural sound can be processed forcrosstalk cancellation and provided through other types of speakers(e.g., dipole stereo speakers).

From the point-of-view of the listener, the sound originates or emanatesfrom the object, point, area, or location that corresponds with the SLP.For example, an example embodiment selects an intended SLP at, on, ornear a physical object, a VR object, or an AR object that is orrepresents the sound source (including locations behind the object orsound source). When the sound is convolved with HRTFs corresponding tothe intended SLP (including HRTFs behind the intended SLP), then thesound appears to originate to the listener at the object.

When binaural sound is provided to the listener, the listener will hearthe sound as if it originates from the sound source. The sound, however,does not originate from the sound source since the sound source may bean inanimate object with no electronics or an animate object with noelectronics. Alternatively, the sound source has electronics but doesnot have the capability to generate sound (e.g., the sound source has nospeakers or sound system). As yet another example, the sound source hasspeakers and the ability to provide sound but is not providing sound tothe listener. In each of these examples, the listener perceives thesound to originate from the sound source, but the sound source does notproduce the sound. Instead, the sound is altered or convolved andprovided to the listener so the sound appears to originate from thesound source.

Sound localization information (SLI) is stored and categorized invarious formats. For example, tables or lookup tables store SLI forquick access and provide convolution instructions for sound. Informationstored in tables expedites retrieval of stored information, reduces CPUtime required for sound convolution, and reduces a number of instructioncycles. Storing SLI in tables also expedites and/or assists inprefetching, preprocessing, caching, and executing other exampleembodiments discussed herein.

Consider an example in which a HPED determines an identity of a listener(e.g., with a biometric sensor such as one discussed herein) andretrieves HRTFs associated with the identified listener. For example, aHPED captures, with a camera in the HPED, the face of a first userduring telephony with a second user. Facial recognition softwareanalyzes the facial image of the first user to determine his or heridentity. Memory stores HRTFs for different users (e.g., personalizedHRTFs or preferred HRTFs). Based on the identity of the user, the HPEDretrieves far-field HRTFs that are assigned to the first user. Aprocessor in the HPED or a process in a server in communication with theHPED convolves the voice of the second user with the selected far-fieldHRTFs. When the camera no longer detects or recognizes the face of thefirst user, the HPED changes the voice of the second user fromlocalizing as the binaural sound to localizing as one of mono sound orstereo sound.

In an example embodiment, the HRTFs and SLI being executed to convolvethe sound can switch or change depending on whether the distance of thelistener to the sound source is near-field or far-field. Consider anexample in which a display displays a sound source to a listener who istwo meters away from the sound source. When the listener is two metersaway from the sound source, the listener is a far-field distance fromthe sound source. The example embodiment convolves the sound withfar-field HRTFs with coordinate locations that correspond to thelocation of the sound source. The coordinate locations of the SLIcorrespond or match the coordinate locations of the sound source. Thelistener then moves closer to the sound source and is within anear-field distance to the sound source, say within 0.5 meters of thesound source. Instead of convolving the sound to the location of thesound source which would require near-field HRTFs, an example embodimentswitches or changes convolution to convolve the sound to a location thatis behind the sound source so far-field HRTFs can continue to be used toconvolve the sound. For example, the example embodiment selectsfar-field HRTFs with a distance of 1.0 meters and adjusts the SLIaccordingly. The sound is actually convolved with SLI to coordinateslocated 1.0 meters away from the listener which would be 0.5 metersbehind the sound source. The listener, though, perceives the sound asoriginating from the sound source even though the coordinate locationsof the HRTFs do not match the coordinate location of the sound source(e.g., here, the HRTFs have a distance of 1.0 meters while the soundsource has a distance of 0.5 meters from the listener). This exampleillustrates one way in which convolution of the sound changes based on adistance of the listener to the sound source. The example embodimentcontinues to convolve the sound with far-field HRTFs even when thelistener is a near-field distance to the sound source.

FIG. 10 is a method that convolves, during a telephone call, a voice ofa sound source to a location behind the sound source in accordance withan example embodiment.

Block 1000 states simultaneously display, with an electronic device, twoor more sound sources to a person during a telephone call.

For example, a display of an HPED simultaneously displays two or moresound sources that represent users to the telephone call. For instance,each person to a conference call or video call is displayed on thedisplay.

For example, a wearable electronic device (e.g., electronic glasses, aHMD, or a smartphone worn on a head of a person) displays an AR or VRimage of each user to the telephone call.

The sound sources include the wearer or holder or user of the electronicdevice. For example, a caller calls a person who wears or holds aportable electronic device, and this electronic device displays an imageof a caller and also an image of the person to the person during thetelephone call.

Block 1010 states convolve and/or process, during the telephone call, atleast one voice of the one sound source being displayed with HRTFshaving coordinate locations located behind the one sound source beingdisplayed.

In an example embodiment, the coordinate location of the HRTFs for asound source is directly behind the sound source. For instance, thecoordinate location is located 0.1 m-1.0 m behind the location of thesound source. Further, the coordinate location is located on or about ona line-of-sight from the person to the sound source or on or near a rayextending from the center of the head of the person to therepresentation of a caller or sound on the telephone call.

The coordinate location of the HRTFs can be located off or away from theray/line-of-sight or at an angle with respect to the ray orline-of-sight. For example, the coordinate location of the HRTFs isangled from the line-of-sight by an azimuth angle (θ) and/or elevationangle (ϕ) of 1°, 2°, 3°, 4°, 5°, 6°, 7°, 8°, 9°, 10°, 15°, 20°, or 25°.

FIG. 11 is a computer system or electronic system 1100 in accordancewith an example embodiment. The computer system includes one or more ofa portable electronic device or PED 1102, one or more computers orelectronic devices (such as one or more servers) 1104, and storage ormemory 1108 in communication over one or more networks 1110.

The portable electronic device 1102 includes one or more components ofcomputer readable medium (CRM) or memory 1120 (such as cache memory andmemory storing instructions to execute one or more example embodiments),a display 1122, a processing unit 1124 (such as one or more processors,microprocessors, and/or microcontrollers), one or more interfaces 1126(such as a network interface, a graphical user interface, a naturallanguage user interface, a natural user interface, a phone controlinterface, a reality user interface, a kinetic user interface, atouchless user interface, an augmented reality user interface, and/or aninterface that combines reality and virtuality), a sound localizationsystem (SLS) 1128, head tracking 1130, a digital signal processor (DSP)1132, and one or more sensors 1134 (such as a camera, proximity sensor,or other sensor discussed herein).

The PED 1102 communicates with wired or wireless headphones or earphones1103 that include speakers 1140 and/or other electronics (such asmicrophones).

The storage 1108 includes one or more of memory or databases that storeone or more of audio files, sound information, sound localizationinformation, audio input, SLPs and/or zones, software applications, userprofiles and/or user preferences (such as user preferences for SLPlocations and sound localization preferences), impulse responses andtransfer functions (such as HRTFs, HRIRs, BRIRs, and RIRs), and otherinformation discussed herein.

The network 1110 includes one or more of a cellular network, a publicswitch telephone network, the Internet, a local area network (LAN), awide area network (WAN), a metropolitan area network (MAN), a personalarea network (PAN), home area network (HAM), and other public and/orprivate networks. Additionally, the electronic devices need notcommunicate with each other through a network. As one example,electronic devices couple together via one or more wires, such as adirect wired-connection. As another example, electronic devicescommunicate directly through a wireless protocol, such as Bluetooth,near field communication (NFC), or other wireless communicationprotocol.

Electronic device 1104 (shown by way of example as a server) includesone or more components of computer readable medium (CRM) or memory 1160(including cache memory), a processing unit 1164 (such as one or moreprocessors, microprocessors, and/or microcontrollers), a soundlocalization system 1166, and an audio or sound convolver 1168.

The electronic device 1104 communicates with the PED 1102 and withstorage or memory that stores sound localization information (SLI) 1180,such as transfer functions and/or impulse responses (e.g., HRTFs, HRIRs,BRIRs, etc. for multiple users) and other information discussed herein.Alternatively or additionally, the transfer functions and/or impulseresponses and other SLI are stored in memory 1120 or another location,such as storage 1108.

FIG. 12 is a computer system or electronic system in accordance with anexample embodiment. The computer system 1200 includes one or more of aportable electronic device 1202, a server 1204, a portable electronicdevice 1208 (including wearable electronic devices and handheld portableelectronic devices), and a display 1205 in communication with each otherover one or more networks 1212.

Portable electronic device 1202 includes one or more components ofcomputer readable medium (CRM) or memory 1220 (including cache memory),one or more displays 1222, a processor or processing unit 1224 (such asone or more microprocessors and/or microcontrollers), one or moresensors 1226 (such as a micro-electro-mechanical systems sensor, aproximity sensor, a biometric sensor, an optical sensor, aradio-frequency identification sensor, a global positioning satellite(GPS) sensor, a solid state compass, a gyroscope, a magnetometer, and/oran accelerometer), earphones with speakers 1228, sound localizationinformation (SLI) 1230, an intelligent user agent (IUA) and/orintelligent personal assistant (IPA) 1232, sound hardware 1234, and aSLP selector 1238.

Server 1204 includes computer readable medium (CRM) or memory 1250, aprocessor or processing unit 1252, and a DSP 1254 and/or other hardwareto convolve audio in accordance with an example embodiment.

Portable electronic device 1208 includes computer readable medium (CRM)or memory 1260 (including cache memory), one or more displays 1262, aprocessor or processing unit 1264, one or more interfaces 1266 (such asinterfaces discussed herein in FIG. 11 ), sound localization information1268 (e.g., stored in memory), a sound localization point (SLP) selector1270, user preferences 1272, one or more digital signal processors (DSP)1274, one or more of speakers and/or microphones 1276, head trackingand/or head orientation determiner 1277, a compass 1278, inertialsensors 1279 (such as an accelerometer, a gyroscope, and/or amagnetometer), and a camera 1280.

A sound localization point (SLP) selector includes specialized hardwareand/or software to execute example embodiments that select a desired SLPfor where binaural sound will localize to a user and/or selectcoordinate locations of HRTFs being executed to convolve the sound.

A sound localization system (SLS) and SLP selector include one or moreof a processor, core, chip, microprocessor, controller, memory,specialized hardware, and specialized software to execute one or moreexample embodiments (including one or more methods discussed hereinand/or blocks discussed in a method). By way of example, the hardwareincludes a customized integrated circuit (IC) or customizedsystem-on-chip (SoC) to select, assign, and/or designate a SLP or acoordinate location for sound or convolve sound with SLI to generatebinaural sound. For instance, an application-specific integrated circuit(ASIC) or a structured ASIC are examples of a customized IC that isdesigned for a particular use, as opposed to a general-purpose use. Suchspecialized hardware also includes field-programmable gate arrays(FPGAs) designed to execute a method discussed herein and/or one or moreblocks discussed herein. For example, FPGAs are programmed to executeselecting, assigning, and/or designating SLPs and coordinate locationsfor sound or convolving, processing, or preprocessing sound so the soundexternally localizes to the listener.

The sound localization system (SLS) performs various tasks with regardto managing, generating, interpolating, extrapolating, retrieving,storing, and selecting SLPs and coordinate locations and can function incoordination with and/or be part of the processing unit and/or DSPs orcan incorporate DSPs. These tasks include, determining coordinates ofSLPs and other coordinate locations and their corresponding HRTFs,switching and/or changing sound between binaural sound and mono sound orstereo sound, selecting SLPs and/or coordinate locations of HRTFs for auser, selecting sound sources to which sound will localize to a user,designating a type of sound, segment of audio, or sound source,providing binaural sound to users at a SLP, prefetching and/orpreprocessing SLI, and executing one or more other blocks discussedherein. The sound localization system can also include a soundconvolving application that convolves and de-convolves sound accordingto one or more audio impulse responses and/or transfer functions basedon or in communication with head tracking.

In an example embodiment, the SLS calculates the line-of-sight orimaginary line from the head of the listener to the sound source andretrieves SLI (including HRTFs) based on the location of the line. Forinstance, two points or locations determine a line. One point is locatedat the head of the listener. Information about the head orientation ofthe listener can be determined from or calculated from a camera or ahead tracking and/or head orientation determiner (e.g., hardware and/orsoftware in a head mounted display or other wearable electronic device).A second point is located at the origin of the sound, such as theelectronic device, sound source, etc. Information about the location ofthe second point can be determined from or calculated from a camera, asensor, tag or RFID, or an electronic device. For instance, anelectronic device calculates its position with respect to a head of thelistener using one or more of a camera, facial recognition, a MEMSsensor (e.g., a multi-axis sensor with 9 degrees of freedom), wirelessshort-range communication with another electronic device (e.g.,communication between an HPED and a wearable electronic device orelectronic device in an Internet-of-Things (IoT) network), or othermethod.

By way of example, an intelligent personal assistant or intelligent useragent is a software agent that performs tasks or services for a person,such as organizing and maintaining information (such as emails,messaging (e.g., instant messaging, mobile messaging, voice messaging,store and forward messaging), calendar events, files, to-do items,etc.), initiating telephony requests (e.g., scheduling, initiating,and/or triggering phone calls, video calls, and telepresence requestsbetween the user, IPA, other users, and other IPAs), responding toqueries, responding to search requests, information retrieval,performing specific one-time tasks (such as responding to a voiceinstruction), file request and retrieval (such as retrieving andtriggering a sound or video to play, or text or images to display),timely or passive data collection or information-gathering from personsor users (such as querying a user for information), data and voicestorage, management and recall (such as taking dictation, storing memos,managing lists), memory aid, reminding of users, performing ongoingtasks (such as schedule management and personal health or financemanagement), and providing recommendations. By way of example, thesetasks or services are based on one or more of user input, prediction,activity awareness, location awareness, an ability to access information(including user profile information and online information), userprofile information, and other data or information.

By way of example, the sound hardware includes a sound card and/or asound chip. A sound card includes one or more of a digital-to-analog(DAC) converter, an analog-to-digital (ATD) converter, a line-inconnector for an input signal from a source of sound, a line-outconnector, a hardware audio accelerator providing hardware polyphony,and one or more digital-signal-processors (DSPs). A sound chip is anintegrated circuit (also known as a “chip”) that produces sound throughdigital, analog, or mixed-mode electronics and includes electronicdevices such as one or more of an oscillator, envelope controller,sampler, filter, and amplifier. The sound hardware can be or includecustomized or specialized hardware that processes and convolves mono andstereo sound into binaural sound.

By way of example, a computer and a portable electronic device include,but are not limited to, handheld portable electronic devices (HPEDs),wearable electronic glasses, smartglasses, watches, wearable electronicdevices (WEDs) or wearables, smart earphones or hearables, voice controldevices (VCD), voice personal assistants (VPAs), network attachedstorage (NAS), printers and peripheral devices, virtual devices oremulated devices (e.g., device simulators, soft devices), cloud residentdevices, computing devices, electronic devices with cellular or mobilephone capabilities, digital cameras, desktop computers, servers,portable computers (such as tablet and notebook computers), smartphones,electronic and computer game consoles, home entertainment systems,digital audio players (DAPs) and handheld audio playing devices(example, handheld devices for downloading and playing music andvideos), appliances (including home appliances), head mounted displays(HMDs), optical head mounted displays (OHMDs), personal digitalassistants (PDAs), electronics and electronic systems in automobiles(including automobile control systems), combinations of these devices,devices with a processor or processing unit and a memory, and otherportable and non-portable electronic devices and systems (such aselectronic devices with a DSP and/or sound hardware as discussedherein).

The SLP selector and/or SLS can also execute retrieving SLI,preprocessing, predicting, and caching including, but not limited to,predicting an action of a user, predicting a location of a user,predicting motion of a user such as a gesture, a change in a headdisplacement and/or orientation, predicting a trajectory of a soundlocalization to a user, predicting an event, predicting a desire or wantof a user, predicting a query of a user (such as a query to or responsefrom an intelligent personal assistant), predicting and/or recommendinga SLP, zone, or RIR/RTF to a user, etc. Such predictions can alsoinclude predicting user actions or requests in the future (such as alikelihood that the user or electronic device localizes a type of soundto a particular SLP or zone). For instance, determinations by a softwareapplication, hardware, an electronic device, and/or user agent aremodeled as a prediction that the user will take an action and/or desireor benefit from moving or muting a SLP, from delaying the playing of asound, from a switch between binaural, mono, and stereo sounds or achange to binaural sound (such as pausing binaural sound, mutingbinaural sound, selecting an object at which to localize sound, reducingor eliminating one or more cues or spatializations or localizations ofbinaural sound). For example, an analysis of historical events, personalinformation, geographic location, and/or the user profile provides aprobability and/or likelihood that the user will take an action (such aswhether the user prefers a particular SLP or zone as the location forwhere sound will localize, prefers binaural sound or stereo, or monosound for a particular location, prefers a particular listeningexperience, or a particular communication with another person or anintelligent personal assistant). By way of example, one or morepredictive models execute to predict the probability that a user wouldtake, determine, or desire the action. The predictor also predictsfuture events unrelated to the actions of the user including, but notlimited to, a prediction of times, locations, or identities of incomingcallers or virtual sound source requests for sound localizations to theuser, a type or quality of inbound sound, predicting a sound source orvirtual sound source path including a change in orientation of the soundsource or virtual sound source or SLP such as a change in a direction ofsource emission of the SLP.

Example embodiments are not limited to HRTFs but also include othersound transfer functions and sound impulse responses including, but notlimited to, head related impulse responses (HRIRs), room transferfunctions (RTFs), room impulse responses (RIRs), binaural room impulseresponses (BRIRs), binaural room transfer functions (BRTFs), headphonetransfer functions (HPTFs), etc.

Examples herein can take place in physical spaces, in computer renderedspaces (such as computer games or VR), in partially computer renderedspaces (AR), and in combinations thereof.

The processor unit includes a processor (such as a central processingunit, CPU, microprocessor, microcontrollers, field programmable gatearrays (FPGA), application-specific integrated circuits (ASIC), etc.)for controlling the overall operation of memory (such as random accessmemory (RAM) for temporary data storage, read-only memory (ROM) forpermanent data storage, and firmware). The processing unit and DSPcommunicate with each other and memory and perform operations and tasksthat implement one or more blocks of the flow diagrams discussed herein.The memory, for example, stores applications, data, programs, algorithms(including software to implement or assist in implementing exampleembodiments) and other data.

Consider an example embodiment in which the SLS includes an integratedcircuit FPGA that is specifically customized, designed, configured, orwired to execute one or more blocks discussed herein. For example, theFPGA includes one or more programmable logic blocks that are wiredtogether or configured to execute combinational functions for the SLS(e.g., changing between binaural sound and mono sound upon detectingrotation of the HPED or detecting another action discussed herein).

Consider an example in which the SLS includes an integrated circuit orASIC that is specifically customized, designed, or configured to executeone or more blocks discussed herein. For example, the ASIC hascustomized gate arrangements for the SLS. The ASIC can also includemicroprocessors and memory blocks (such as being a SoC (system-on-chip)designed with special functionality to execute functions of the SLSand/or blocks of methods discussed herein).

Consider an example in which the SLS includes one or more integratedcircuits that are specifically customized, designed, or configured toexecute one or more blocks discussed herein. For example, the electronicdevices include a specialized or customized processor or microprocessoror semiconductor intellectual property (SIP) core or digital signalprocessor (DSP) with a hardware architecture optimized for convolvingsound and executing one or more example embodiments.

Consider an example in which the HPED includes a customized or dedicatedDSP that executes one or more blocks discussed herein (includingprocessing and/or convolving sound into binaural sound). Such a DSP hasa better power performance or power efficiency compared to ageneral-purpose microprocessor and is more suitable for a HPED, such asa smartphone, due to power consumption constraints of the HPED. The DSPcan also include a specialized hardware architecture, such as a specialor specialized memory architecture to simultaneously fetch or prefetchmultiple data and/or instructions concurrently to increase executionspeed and sound processing efficiency. By way of example, streamingsound data (such as sound data in a telephone call or software gameapplication) is processed and convolved with a specialized memoryarchitecture (such as the Harvard architecture or the Modified vonNeumann architecture). The DSP can also provide a lower-cost solutioncompared to a general-purpose microprocessor that executes digitalsignal processing and convolving algorithms. The DSP can also providefunctions as an application processor or microcontroller.

Consider an example in which a customized DSP includes one or morespecial instruction sets for multiply-accumulate operations (MACoperations), such as convolving with transfer functions and/or impulseresponses (such as HRTFs, HRIRs, BRIRs, et al.), executing Fast FourierTransforms (FFTs), executing finite impulse response (FIR) filtering,and executing instructions to increase parallelism.

Consider an example in which the DSP includes the SLP selector. Forexample, the SLP selector and/or the DSP are integrated onto a singleintegrated circuit die or integrated onto multiple dies in a single chippackage to expedite binaural sound processing.

Consider another example in which HRTFs (such as a custom or personalset of HRTFs created for a certain user or users, or other transferfunctions or impulse responses) are stored or cached in the DSP memoryor local memory relatively close to the DSP to expedite binaural soundprocessing.

Consider an example in which a smartphone or other PED includes one ormore dedicated sound DSPs (or dedicated DSPs for sound processing, imageprocessing, and/or video processing). The DSPs execute instructions toconvolve sound and display locations of images or SLPs for the sound ona user interface of a HPED. Further, the DSPs simultaneously convolvemultiple sound sources or SLPs to a user. These sound sources or SLPscan be moving with respect to the face of the user so the DSPs convolvemultiple different sound signals and virtual sound sources with HRTFsthat are continually, continuously, or rapidly changing.

As used herein, the word “about” when indicated with a number, amount,time, etc. is close or near something. By way of example, for sphericalor polar coordinates of a SLP (r, θ, ϕ), the word “about” means plus orminus (±) three degrees for 0 and ϕ and plus or minus 5% for distance(r).

As used herein, “empty space” is a location that is not occupied by atangible object.

As used herein, “field-of-view” is the observable world that is seen ata given moment. Field-of-view includes what a user or camera sees in avirtual or augmented world (e.g., what the user sees while wearing a HMDor OHMD).

As used herein, “line-of-sight” is a line from an observer's eye to alocation.

As used herein, “proximate” means near. For example, a sound thatlocalizes proximate to a listener occurs within two meters of theperson.

As used herein, “sound localization information” or “SLI” is informationthat an electronic device uses to process or convolve sound so the soundexternally localizes as binaural sound to a listener. Examples of SLIinclude head related transfer functions (HRTFs), head related impulseresponses (HRIRs), binaural room impulse responses (BRIRs), room impulseresponses (RIRs), interaural level differences (ILDs), and interauraltime differences (ITDs).

As used herein, a “sound localization point” or “SLP” is a locationwhere a listener localizes sound. A SLP can be internal (such asmonaural sound that localizes inside a head of a listener wearingheadphones or earbuds), or a SLP can be external (such as binaural soundthat externally localizes to a point or an area that is away from butproximate to the person or away from but not near the person). A SLP canbe a single point such as one defined by a single pair of HRTFs or a SLPcan be a zone or shape or volume or general area, such as a line or acylindrical volume. Further, in some instances, multiple impulseresponses or transfer functions can process or convolve sounds to aplace within the boundary of the SLP. In some instances, HRTFs necessaryto produce a particular SLP for a particular user may not have beencreated. A HRTF may not be required to provide a SLP or localize soundfor a user, such as for an internalized SLP, or a SLP may be rendered byadjusting an ITD and/or ILD or other human audial cues.

A “sound source” and a “source of sound” are interchangeable and are areal or virtual object or location to where a listener localizesbinaural sound, such as an object to which a listener externallylocalizes binaural sound. Examples include, but are not limited to, anelectronic device, an image, a physical or real or tangible object, avirtual object or VR image, a video, a picture, an AR image, a virtualsound source, a display, a location from where a listener is intended tolocalize binaural sound, a combination of one or more of these examples,and other examples provided herein.

As used herein, “spherical coordinates” or “spherical coordinate system”provides a coordinate system in 3D space in which a position is givenwith three numbers: a radial distance (r) from an origin, an azimuthangle (θ) of its orthogonal projection on a reference plane that isorthogonal to the zenith direction and that passes through the origin,and an elevation or polar angle (ϕ) that is measured from the zenithdirection.

As used herein, a “telephone call,” or a “phone call” or “telephony” isa connection over a wired and/or wireless network between a callingperson or user and a called person or user. Telephone calls can uselandlines, mobile phones, satellite phones, HPEDs, voice personalassistants (VPAs), computers, and other portable and non-portableelectronic devices. Further, telephone calls can be placed through oneor more of a public switched telephone network, the internet, andvarious types of networks (such as Wide Area Networks or WANs, LocalArea Networks or LANs, Personal Area Networks or PANs, home areanetworks or HAMs, Campus Area Networks or CANs, etc.). Telephone callsinclude other types of telephony including Voice over Internet Protocol(VoIP) calls, video calls, conference calls, internet telephone calls,in-game calls, telepresence, etc.

As used herein, “three-dimensional space” or “3D space” is space inwhich three values or parameters are used to determine a position of anobject or point. For example, binaural sound can localize to locationsin 3D space around a head of a listener. 3D space can also exist invirtual reality (e.g., a user wearing a HMD can see a virtual 3D space).

As used herein, a “user” or a “listener” is a person (i.e., a humanbeing). These terms can also be a software program (including an IPA orIUA), hardware (such as a processor or processing unit), an electronicdevice or a computer (such as a speaking robot or avatar shaped like ahuman with microphones or points of virtual microphones in or at itsears).

As used herein, a “video call” is a telephone call in which one or morepeople to the video call see video of the other person.

As used herein, a “virtual sound source” is a sound source in virtualauditory space (aka virtual acoustic space). For example, listeners heara virtual sound source at one or more SLPs.

Impulse responses can be transformed into their respective transferfunctions. For example, a RIR has an equivalent transfer function of aRTF; a BRIR has an equivalent transfer function of a BRIR; and a HRIRhas an equivalent transfer function of a HRTF.

In some example embodiments, the methods illustrated herein and data andinstructions associated therewith, are stored in respective storagedevices that are implemented as computer-readable and/ormachine-readable storage media, physical or tangible media, and/ornon-transitory storage media. These storage media include differentforms of memory including semiconductor memory devices such as NANDflash non-volatile memory, DRAM, or SRAM, Erasable and ProgrammableRead-Only Memories (EPROMs), Electrically Erasable and ProgrammableRead-Only Memories (EEPROMs), solid state drives (SSD), and flashmemories; magnetic disks such as fixed and removable disks; othermagnetic media including tape; optical media such as Compact Disks (CDs)or Digital Versatile Disks (DVDs). Note that the instructions of thesoftware discussed above can be provided on computer-readable ormachine-readable storage medium, or alternatively, can be provided onmultiple computer-readable or machine-readable storage media distributedin a large system having possibly plural nodes. Such computer-readableor machine-readable medium or media is (are) considered to be part of anarticle (or article of manufacture). An article or article ofmanufacture can refer to a manufactured single component or multiplecomponents.

Blocks and/or methods discussed herein can be executed and/or made by auser, a user agent (including machine learning agents and intelligentuser agents), a software application, an electronic device, a computer,firmware, hardware, a process, a computer system, and/or an intelligentpersonal assistant. Furthermore, blocks and/or methods discussed hereincan be executed automatically with or without instruction from a user.

The methods in accordance with example embodiments are provided asexamples, and examples from one method should not be construed to limitexamples from another method. Tables and other information show exampledata and example structures; other data and other database structurescan be implemented with example embodiments. Further, methods discussedwithin different figures can be added to or exchanged with methods inother figures. Further yet, specific numerical data values (such asspecific quantities, numbers, categories, etc.) or other specificinformation should be interpreted as illustrative for discussing exampleembodiments. Such specific information is not provided to limit exampleembodiments.

What is claimed is:
 1. A method, comprising: displaying, with a wearableelectronic device (WED) worn on a head a user, a virtual object thatappears within a near-field distance from the user; and processing, withone or more processors in the WED, sound for the virtual object togenerate binaural sound that has a sound localization point (SLP) with acoordinate location at a far-field distance from the user that occursalong a line-of-sight from the user to the virtual object and behind thevirtual object while the virtual object is within the near-fielddistance from the user.
 2. The method of claim 1 further comprising:tracking, with one or more sensors in the WED, movement of the head ofthe user; and processing, with the one or more processors and while thehead of the user moves with respect to the virtual object, the sound forthe virtual object such that the SLP remains behind the virtual objectwhile the virtual object remains within the near-field distance.
 3. Themethod of claim 1 further comprising: continuing to process, with theone or more processors, the sound for the virtual object with far-fieldhead-related transfer functions (HRTFs) while the sound emanates fromthe virtual object with the virtual object located within the near-fielddistance from the user.
 4. The method of claim 1 further comprising:sensing, with one or more sensors in the WED, movement of the WED whileworn on the head of the user; and automatically lowering a volume of thesound in response to the one or more sensors sensing the movement. 5.The method of claim 1, wherein the one or more processors include adigital signal processor (DSP) that processes the sound for the virtualobject that is displayed within the near-field distance from the userwith far-field head-related transfer functions (HRTFs) as long as thesound originates from the virtual object with the user located withinthe near-field distance from the virtual object.
 6. The method of claim1, wherein the coordinate location of the SLP is located a distance (d1)from the head of the user in which d1>1.0 meter, the virtual object islocated a distance (d2) from the head of the user in which d2<1.0 meter,the WED is a head mounted display (HMD), and the virtual object occursin virtual reality (VR).
 7. The method of claim 1 further comprising:sensing, with one or more sensors in the WED, movement of the WED whileworn on the head of the user; and automatically muting the sound inresponse to the one or more sensors sensing the movement.
 8. A wearableelectronic device (WED) worn on a head of a user, the WED comprising: adisplay that displays a virtual image that occurs within a near-fielddistance from the head of the user; and a digital signal processor (DSP)that processes sound for the virtual image with far-field head-relatedtransfer functions (HRTFs) to generate binaural sound that has a soundlocalization point (SLP) that occurs at a far-field distance locatedbehind the virtual image and along a line-of-sight from the user to thevirtual image while the virtual image is located within a near-fielddistance from the head of the user.
 9. The WED of claim 8, wherein thevirtual image is an augmented reality (AR) image, and the display isincluded with electronic glasses worn on the head of the user.
 10. TheWED of claim 8, wherein the virtual image is a virtual reality (VR)image, and the display is included with a head mounted display (HMD)worn on the head of the user.
 11. The WED of claim 8 further comprising:one or more sensors that sense movement of the WED while the WED is wornon the head of the user, wherein the WED lowers a volume of the sound inresponse to the one or more sensors sensing the movement.
 12. The WED ofclaim 8 further comprising: one or more sensors that sense movement ofthe WED while the WED is worn on the head of the user, wherein the WEDmutes the sound in response to the one or more sensors sensing themovement.
 13. The WED of claim 8 further comprising: one or more sensorsthat sense movement of the WED while the WED is worn on the head of theuser, wherein the WED terminates an electronic call in response to theone or more sensors sensing the movement.
 14. The WED of claim 8,wherein the virtual image is an augmented reality (AR) image of anintelligent user assistant (IUA) that is located less than one meteraway from the head of the user, and a voice of the IUA occurs at acoordinate location behind the AR image of the IUA when the AR image iswithin the near-field distance from the head of the user.
 15. The WED ofclaim 8, wherein the virtual image is a virtual reality (VR) image of anintelligent user assistant (IUA) that is located less than one meteraway from the head of the user, and a voice of the IUA occurs at acoordinate location behind the VR image of the IUA when the VR image iswithin the near-field distance from the head of the user.
 16. A wearableelectronic device (WED) worn on a head of a user, the WED comprising: adisplay that displays a virtual image that is located within anear-field distance from the head of the user viewing the virtual image;and one or more processors that process sound for the virtual image withhead-related transfer functions (HRTFs) to generate binaural sound thathas a sound localization point (SLP) with a coordinate location thatoccurs at a far-field distance located behind the virtual image along aline-of-sight from the user to the virtual image while the displaydisplays the virtual image to the user at the near-field distance,wherein the coordinate location of the binaural sound continues to occurat the far-field distance located behind the virtual image when the headof the user moves while the virtual image remains within the near-fielddistance from the head of the user.
 17. The WED of claim 16, wherein thedisplay is an augmented reality (AR) display worn on the head of theuser viewing the virtual image, and the virtual image is an AR image.18. The WED of claim 16, wherein the display is a virtual reality (VR)display worn on the head of the user viewing the virtual image, and thevirtual image is a VR image.
 19. The WED of claim 16 further comprising:one or more sensors that sense a direction of gaze of the user, whereinthe WED mutes the sound in response to sensing that the direction ofgaze is not directed to the virtual image.
 20. The WED of claim 16further comprising: one or more sensors that sense a direction of gazeof the user, wherein the WED switches the sound to mono sound or stereosound in response to sensing that the direction of gaze is not directedto the virtual image.